| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Commit 155f0c3f28a1 ("maildir/sort: get MessageInfos in parallel") added
parallel parsing of maildir messages. The parsing is limitted to 2x the
number of CPUs using a buffered channel. The channel is drained at the
end of the parsing, however it is not drained if a parse error is
encountered. With enough errors, this can lead to a deadlock.
Drain the channel when an error occurs.
Fixes: 155f0c3f28a1 ("maildir/sort: get MessageInfos in parallel")
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Commit 29205fdd07c0 ("maildir/search: get MessageInfos in parallel")
introduced parallel searching of maildir backends, which improves search
and filtering performance. When no filter is specified, the search
algorithm is still used with no criteria because the FilterCriteria
field always contains "filter" as a minimum. With the new parallelized
search, the message order becomes random. Only use the search algorithm
if there is a filter specified.
If a filter *is* specified and a sort order is *not*, the messages will
go through the search algorithm and become randomized, but not sorted
back in order. Always default to a UID order sort (the same as prior to
this commit) when no sort order is specified.
Fixes: 29205fdd07c0 ("maildir/search: get MessageInfos in parallel")
Reported-by: Ben Lee-Cohen <bdc@fastmail.fm>
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Tested-by: Ben Lee-Cohen <bdc@fastmail.fm>
|
|
|
|
|
|
|
|
|
|
| |
Add thread capability to maildir backend. Enables the maildir worker to
thread the entire folder, as opposed to only the displayed messages via
client side threads.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Tested-by: Inwit <inwit@sindominio.net>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
| |
When moving or archiving to a very large folder, aerc requests directory
info, which can takes several seconds to receive. Also, when moving, we
post a MessagesDeleted message. Remove unnecessary call to directory
info and change posted message to the correct MessagesMoved.
Signed-off-by: Bence Ferdinandy <bence@ferdinandy.com>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Sorting opens and reads portions of every file within a directory in
order to gather the data needed. Specifically, RFC822Headers and
BodyStructure are not needed. The RFC822Headers field stores a lot of
information, and the BodyStructure field requires parsing the body of
the email.
Don't set these two values when parsing.
Note: in my testing, this dropped sorting a 52k archive from 2.2gb of
ram usage, to < 500mb
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
| |
Searching in the maildir worker requires reading each file in the
directory. Use waitgroups and goroutines to read files in parallel.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
| |
Sorting in the maildir worker requires reading each file in the
directory. Use waitgroups and goroutines to read files in parallel.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The maildir worker's check-mail command did not include the SkipSort
directive in the DirectoryInfo, which would cause a fresh fetching of
directory contents on the selected maiboxes. This is unneeded and
resulted in every directory being sent to the msgstore and requesting a
sort.
Set SkipSort when checking mail to prevent extra calls to sort.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
| |
Replace ListFolder with a new method that returns a map indexed by
folder names instead of a list of folder names. A map is simpler to use
and more efficient in case we only want to check the presence of a
specific folder in the Maildir store.
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
|
|
|
|
|
|
|
|
|
| |
This change moves code that could be common to both notmuch and maildir
workers in worker/lib.
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previous change "maildir: hide invalid folders" prevents filepath.Walk
from descending into directories that don't contain new/, tmp/ or cur/
because the WalkDirFunc returns filepath.SkipDir.
We have to assume that these directories can be the parent of valid
maildir folders even if they're not themselves valid maildir folders, so
it is a mistake not to recurse into these subfolders. Fix it by
returning nil instead of filepath.SkipDir in the WalkDirFunc.
Fixes: 150aa0f498b9 ("maildir: hide invalid folders")
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
| |
Non-selected folders will now have their total/unread/new counts updated
in the background when a check-mail happens.
Signed-off-by: Ben Cohen <ben@bencohen.net>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The maildir worker currently populates the list of mail folders by
listing all the filesystem subdirectories in the maildir directory.
Although there's no official specification for maildir subfolders,
they should all have cur/ new/ and tmp/ subdirectories to be valid.
This patch prevents directories that don't have those subdirectories
present on the filesystem from appearing in the account folder list.
This is useful for example to prevent ".notmuch" and ".notmuch/xapian"
from showing up in the folder list if using notmuch to index emails
while using aerc's maildir backend.
Signed-off-by: Julian Pidancet <julian.pidancet@oracle.com>
Acked-by: Tim Culverhouse <tim@timculverhouse.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Subsitute the format specifier %w for %v in the logging facility. The
logging functions use a fmt.Sprintf call behind the scene which does not
recognize %w. %w should be used in fmt.Errorf when you want to wrap
errors. Hence, the log entries that use %w are improperly formatted like
this:
ERROR 2022/10/02 09:13:57.724529 worker.go:439: could not get message
info %!w(*fmt.wrapError=&{could not get structure: [snip] })
^
Links: https://go.dev/blog/go1.13-errors
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Moritz Poldrack <moritz@poldrack.dev>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When an error is encountered fetching a header, the backends respond
with a type.Error worker message. On receipt of this message, the UI
deletes all pending headers. The headers are all requested again as they
remain on the screen, resulting in an infinite request loop - and an
infinite logging loop. The user only ever sees the spinner unless they
check the logs.
A previous commit intended to fix this, however it introduced a
regression where any message that was part of the fetch request would
also be marked as erroneous. This commit is reverted with commit
2aad2fea7d36 ("msgstore: revert 9fdc7acf5b48").
Send an erroneous message info message from the backend when an error is
encountered for a specific UID.
Fixes: 01f80721e283 ("msgstore: post MessageInfo on erroneous fetch")
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
| |
Do not throw an error when the charset is unknown; the message entity
can still be read, but log the error instead.
Reported-by: falsifian
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The maildir worker watches the file system for events (new mail, moved
mail, copied mail, etc). On an event, the worker sends a
DirectoryContents message and a DirectoryInfo message.
Do not send DirectoryContents on FS Event in the maildir worker. The
DirectoryInfo message already triggers the UI to request the new
DirectoryContents, and does so in a more predictable way (IE any
currently applied filter is applied).
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The built-in maildir.Dir.Move method performs an OS level file rename,
which allows for preserving file creation time. Commit c98f70487417
("move: enable MoveMessages from msgstore") enabled the use of the Move
method for maildir. One particular maildir synchronizer (isync/mbsync)
encodes the UID within the filename of the email and cannot recover if
the UID is preserved during a move. mbsync encodes filenames like so:
/path/to/email/{maildir-key},U={uid}:2,S
OfflineIMAP encodes the UID within the filename, but also encodes a hash
of the originating folder so that it can recover from a move without a
rename of the underlying file. OfflineIMAP encodes like so:
/path/to/email{maildir-key},U={uid},FMD5={folder-hash}:2,S
Remove encoded UIDs of the form `,U={uid}` from filenames to prevent
sync issues.
Fixes: https://todo.sr.ht/~rjarry/aerc/75
Fixes: c98f70487417 ("move: enable MoveMessages from msgstore")
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Implement MoveMessages in the maildir worker. go-maildir supports Move
operations by default, and is functionally equivalent to a OS-level
rename. Creation date of the file is preserved by using Move, which is
used by at least one maildir-to-IMAP synchronizer (isync/mbsync). The
previous move method of copy-and-delete would reset the creation date of
the message, and potentially cause sorting issues in other email
clients.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
| |
Since the minimum required version of Go has been bumped to 1.16, the
deprecation of io/ioutil can now be acted upon. This Commit removes the
remaining dependencies on ioutil and replaces them with their io or os
counterparts.
Signed-off-by: Moritz Poldrack <git@moritz.sh>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
| |
Error wrapping as introduced in Go 1.13 adds some additional logic to
use for comparing errors and adding information to it.
Signed-off-by: Moritz Poldrack <moritz@poldrack.dev>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Apply GoDoc comment policy (comments for humans should have a space
after the //; machine-readable comments shouldn't)
Use strings.ReplaceAll instead of strings.Replace when appropriate
Remove if/else chains by replacing them with switches
Use short assignment/increment notation
Replace single case switches with if statements
Combine else and if when appropriate
Signed-off-by: Moritz Poldrack <moritz@poldrack.dev>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
| |
Run `make fmt`.
Signed-off-by: Moritz Poldrack <git@moritz.sh>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
| |
Send error message to UI if check-mail-cmd is required but not set.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
|
| |
Do not pass logger objects around anymore. Shuffle some messages to make
them consistent with the new logging API. Avoid using %v when a more
specific verb exists for the argument types.
The loggers are completely disabled (i.e. Sprintf is not even called)
by default. They are only enabled when redirecting stdout to a file.
Signed-off-by: Robin Jarry <robin@jarry.cc>
Acked-by: Moritz Poldrack <moritz@poldrack.dev>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Append all messages from an mbox file to the selected folder with the
import-mbox command.
User confirmation is required when the folder already contains messages.
A failed append will be retried a few times. If a backend timeout
occurs, the entire import is stopped to prevent a hang.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
| |
Export all message in the current folder to an mbox file. If an error
occurs during the export, aerc retries a few times before giving up to
prevent a hang.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch enables the filtering of a threaded view which uses
server-built threads. Filtering is done server-side, in order to
preserve the use of server-built threads.
In adding this feature, the filtering of notmuch folders was brought up
to feature parity with the other workers. The filters function the same
(ie: they can be stacked). The notmuch filters, however, still use
notmuch syntax for the filtering.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
| |
This patch provides a method to report backend capabilities to the UI.
The intial capabilities included in the report are Sort and Thread.
Having these available to the UI enables the client to better handle
server side threading.
Signed-off-by: Koni Marti <koni.marti@gmail.com>
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
| |
See https://www.courier-mta.org/maildir.html#maildircontents
Signed-off-by: Adnan Maolood <me@adnano.co>
Acked-by: Koni Marti <koni.marti@gmail.com>
|
|
|
|
|
|
|
|
| |
The maildir worker was adding Recent messages to the counts of Unseen
and Exists, however these messages were already included in those counts.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When moving/copying/deleting/archiving a message in imap, the RUE counts
displayed in the dirlist would not update properly. Maildir has (had) an
implementation that recounts the entire directory and updates the
DirectoryInfo after one of these actions.
This patch implements a more efficient method of updating, and also
enables it to apply to IMAP without any additional requests. Upon
completion of the action, the counts are manually updated with the count
of messages in the action and recent and/or unseen states of those
messages. This is more efficient for maildir, because we aren't counting
everything in the store. For IMAP, we get the updates for free because
we are only performing the update after confirmation from the server
that the action has happened.
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Check for new mail (recent, unseen, exists counts) with an external
command, or for imap with the STATUS command, at start or on
reconnection and every X time duration
IMAP:
The selected folder is skipped, per specification. Additional config
options are included for including/excluding folders explicitly.
Maildir/Notmuch:
An external command will be run in the background to check for new mail.
An optional timeout can be used with maildir/notmuch. Default is 10s
New account options:
check-mail
check-mail-cmd (maildir/notmuch only)
check-mail-timeout (maildir/notmuch only), default 10s
check-mail-include (IMAP only)
check-mail-exclude (IMAP only)
If unset, or set less than or equal to 0, check-mail will be ignored
Signed-off-by: Tim Culverhouse <tim@timculverhouse.com>
Tested-by: Moritz Poldrack <moritz@poldrack.dev>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
| |
This commit fixes all occurrences of the abovementioned lint-error in
the codebase.
Signed-off-by: Moritz Poldrack <git@moritz.sh>
Acked-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix a data race due dirInfo pointer being read in the main goroutine
by NewMessageStore and written in the anonymous goroutine launched in
Worker.getDirectoryInfo.
To address the issue raised in https://todo.sr.ht/~rjarry/aerc/16, we
use readdir(3) once, parse and cache its results, this replaces
go-maildir library Dir.Flags based stat(3) and filepath.Glob
causing the issue when N (emails) is large.
Signed-off-by: wagner riffel <w@104d.net>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since commit 01c96e78dfe8 ("Update DirectoryInfo handling for maildir"),
flags are checked for every message of a folder when entering it.
Iterating over all messages of a folder takes a long time for large
collections of emails.
Only count the number of messages and state that the directory info
counts are not accurate. Defer the parsing of message flags in
a goroutine to have a more responsive UI.
Fixes: https://todo.sr.ht/~rjarry/aerc/16
Suggested-by: Koni Marti <koni.marti@gmail.com>
Signed-off-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
| |
This fixes piped full message (:pipe -m) being empty.
Fixes: 904ffacb0e52 ("maildir,notmuch: avoid leaking open files")
Signed-off-by: Nguyễn Gia Phong <mcsinyx@disroot.org>
|
|
|
|
|
|
|
|
|
|
| |
Previously, Message.NewReader returned the wrapped buffered reader
without a reference to the opened file, so the files descriptors
were left unclosed after reading. Now, the file reader is returned
directly and closed on the call site. Buffering is not needed here
because it is an implementation detail of go-message.
Fixes: https://todo.sr.ht/~rjarry/aerc/9
|
|
|
|
|
|
|
|
|
|
| |
Messages flags can also be changed without changing directories.
Changing flags in maildirs means renaming the message files. Also take
renames into account.
Link: https://cr.yp.to/proto/maildir.html
Fixes: f4d3c8fc77f9 ("maildir: watch for external changes")
Signed-off-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When a maildir is synchronized by an external process while aerc is
running (e.g. mbsync), some emails may be moved out of "new" to "cur" or
completely deleted.
These deletions are ignored and aerc may assume these messages are still
here, leading to errors.
Take file deletions into account. Also, add "cur" to the watched
folders since these can be modified as well.
Signed-off-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
| |
I'm not sure what are the implications but it seems required.
Link: https://github.com/golang/go/issues/20883
Signed-off-by: Robin Jarry <robin@jarry.cc>
|
|
|
|
|
|
|
|
| |
In the maildir worker we manually need to track the Recent flag in order for the
notification command etc to work.
Push that responsibility to the container, we must make sure to manually add the
flag though if one grabs the message info.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There was some bug which could be worked around by double emitting an event.
However that proofed to be brittle:
We send the first message here from the worker goroutine:
https://git.sr.ht/~sircmpwn/aerc/tree/a5553438/item/worker/maildir/worker.g=
o#L306
Then Tick() is waked in the main goroutine and calls ProcessMessage:
https://git.sr.ht/~sircmpwn/aerc/tree/a5553438/item/widgets/account.go#L100
ProcessMessage in the main goroutine reads types.Message state with
msg.getId() and msg.InResponseTo():
https://git.sr.ht/~sircmpwn/aerc/tree/a5553438/item/worker/types/worker.go#=
L74-76
Meanwhile in the worker goroutine we call PostMessage for a second
time with a pointer that points to the *same* previous message that
ProcessMessage is reading:
https://git.sr.ht/~sircmpwn/aerc/tree/a5553438/item/worker/maildir/worker.g=
o#L306
The second PostMessage call makes writes to message while
ProcessMessage in the main goroutine is possibly reading:
https://git.sr.ht/~sircmpwn/aerc/tree/a5553438/item/worker/types/worker.go#=
L59
This led to a data race in the event loop
Reported-By: Wagner Riffel <w@104d.net>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The `:rmdir` command removes the current directory (`-f` is required if
the directory is not empty).
This is not supported on the notmuch backend.
An issue with the maildir backend is that some sync programs (e.g.
offlineimap) may recover the directory after it is deleted. They need
to specifically be configured to accept deletions, or special commands
need to be executed (e.g. `offlineimap --delete-folder`) to properly
delete folders.
A danger of using this on the IMAP backend is that it is possible for a
new message to be added to the directory and for aerc to not show it
immediately (due to a slow connection) - using `:rmdir` at this moment
(with `-f` if the directory already contains messages) would delete the
directory and the new message that just arrived (and all other
contents). This is documented in aerc(1) so that users are aware of
possible risks.
|
|
|
|
|
|
|
|
| |
If accounts.conf contains an invalid maildir url, return a nice
error instead of panicking.
Log a couple of different error cases to provide extra
information about the error to the user.
|
|
|
|
|
|
|
| |
Aerc usually used the path []int{1} if it didn't know what the proper path is.
However this only works for multipart messages and breaks if it isn't one.
This patch removes all the hard coding and extracts the necessary helpers to lib.
|
|
|
|
|
|
|
| |
Provide search and filter with the option to specify more flag based
conditions.
Use '-x <flag>' to search for messages with a flag (seen, answered,
flagged) and '-X <flag>' to search for messages without a flag.
|
|
|
|
|
| |
The directory info only needs to be sent once for all the messages, this
reduces unnecessary messages being sent which could lock up the ui.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
More mail flags can now be set, unset, and toggled, not just the
read/seen flag.
This functionality is implemented with a new `:flag` and `:unflag`
command, which are extensions to the matching `:read` and `:unread`
commands, adding support for different flags. In fact, the
`read`/`unread` commands are now recognized aliases to `flag`/`unflag`.
The new commands are also well documented in aerc(1).
The change mostly extends the previous read/unread setting functionality
by adding a selection for the flag to change.
|