aboutsummaryrefslogtreecommitdiffstats
path: root/worker/imap/fetch.go
Commit message (Collapse)AuthorAgeFilesLines
* treewide: replace uint32 uids with opaque stringsRobin Jarry2024-08-281-11/+11
| | | | | | | | | | | | | | | | | Add a new models.UID type (an alias to string). Replace all occurrences of uint32 being used as message UID or thread UID with models.UID. Update all workers to only expose models.UID values and deal with the conversion internally. Only IMAP needs to convert these to uint32. All other backends already use plain strings as message identifiers, in which case no conversion is even needed. The directory tree implementation needed to be heavily refactored in order to accommodate thread UID not being usable as a list index. Signed-off-by: Robin Jarry <robin@jarry.cc> Tested-by: Inwit <inwit@sindominio.net> Tested-by: Tim Culverhouse <tim@timculverhouse.com>
* imap: report errors from serverRobin Jarry2024-07-011-21/+31
| | | | | | | | | | | | Avoid eternal spinner on the message list when the imap server advertises some message UIDs but fails to provide their headers when aerc asks from them. When an error occurs, or if some UIDs are not returned, make sure to report the errors to the message list UI. Signed-off-by: Robin Jarry <robin@jarry.cc> Reviewed-by: Tristan Partin <tristan@partin.io>
* log: move package to libRobin Jarry2024-02-141-1/+1
| | | | | | | This has nothing to do at the root of the source tree. Signed-off-by: Robin Jarry <robin@jarry.cc> Acked-by: Bence Ferdinandy <bence@ferdinandy.com>
* imap: show parseable messages on errorMoritz Poldrack2023-12-301-12/+13
| | | | | | | | | | | | | Currently if there are mails aerc can not properly handle, listing the mailbox is aborted. This is especially bothersome if the message one is looking for would be parseable without issue. Show an error if parsing of a message fails, but handle all other messages as normal. This leads to those messages showing up as still loading. Signed-off-by: Moritz Poldrack <git@moritz.sh> Acked-by: Robin Jarry <robin@jarry.cc>
* imap: implement cancellation of searching and fetchingTim Culverhouse2023-06-201-0/+12
| | | | | | | | | Check for cancelled contexts before and after performing headers or flag fetches and any directory searching. Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Tested-by: Bence Ferdinandy <bence@ferdinandy.com> Acked-by: Robin Jarry <robin@jarry.cc>
* imap: use named loggerRobin Jarry2023-06-121-4/+4
| | | | | | | | Use the worker's logging functions to have all imap related messages prefixed by the account name. Signed-off-by: Robin Jarry <robin@jarry.cc> Tested-by: Inwit <inwit@sindominio.net>
* headers: enable partial header fetchingTim Culverhouse2023-05-161-4/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | Enable partial header fetching by creating config values for headers to specifically include, or specifically exclude. The References field will always be fetched, regardless of the include list. Envelope data is always fetched, but is not shown with :toggle-headers, since it isn't in the RFC822 struct unless explicitly included in the list. Partial headers can break the cache on changes. Update the cache tag key to include the state of the partially-fetched headers. Partial header fetching can have a significant performance increase for IMAP, and for all backends a resource improvement. Some data to support this is below. Gathered by opening aerc, selecting a mailbox with approximately 800 messages and scrolling to the end. Received measured with nethogs, RAM from btop Received | RAM ------------------------------------- All Headers | 9,656 kb | 103 MB Minimum Headers | 896 kb | 36 MB Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Acked-by: Moritz Poldrack <moritz@poldrack.dev> Acked-by: Robin Jarry <robin@jarry.cc>
* messageinfo: report message sizesKoni Marti2023-04-261-0/+2
| | | | | | | Report sizes of the message across all backends. Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
* parse msg-id lists more liberallyNguyễn Gia Phong2023-03-261-4/+2
| | | | | | | | | | | | | | | Some user agents deliberately generate non-standard message identifier lists in In-Reply-To and References headers. Instead of failing silently, aerc now falls back to a desperate parser scavaging whatever looking like <id-left@id-right>. As the more liberal parser being substituted with, References header are now stored for IMAP not only when there's a parsing error. Fixes: 31d2f5be3cec ("message-info: add explicit References field") Signed-off-by: Nguyễn Gia Phong <mcsinyx@disroot.org> Acked-by: Robin Jarry <robin@jarry.cc>
* logging: rename package to logRobin Jarry2022-12-021-6/+6
| | | | | | | | | | Use the same name than the builtin "log" package. That way, we do not risk logging in the wrong place. Suggested-by: Tim Culverhouse <tim@timculverhouse.com> Signed-off-by: Robin Jarry <robin@jarry.cc> Tested-by: Bence Ferdinandy <bence@ferdinandy.com> Acked-by: Tim Culverhouse <tim@timculverhouse.com>
* logging: homogenize levelsRobin Jarry2022-12-021-3/+3
| | | | | | | | | | | | | | | | | | The main goal is to ensure that by default, the log file (if configured) does not grow out of proportions. Most of the logging messages in aerc are actually for debugging and/or trace purposes. Define clear rules for logging levels. Enforce these rules everywhere. After this patch, here is what the log file looks like after starting up with a single account: INFO 2022/11/24 20:26:16.147164 aerc.go:176: Starting up version 0.13.0-100-g683981479c60 (go1.18.7 amd64 linux) INFO 2022/11/24 20:26:17.546448 account.go:254: [work] connected. Signed-off-by: Robin Jarry <robin@jarry.cc> Tested-by: Bence Ferdinandy <bence@ferdinandy.com> Acked-by: Tim Culverhouse <tim@timculverhouse.com>
* imap: fix error with empty flag updatesRobin Jarry2022-11-231-4/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix the following error occurring when fetching new headers after IDLE: panic: runtime error: invalid memory address or nil pointer dereference bufio.(*Reader).fill(0xc00029af00) bufio/bufio.go:106 +0xd8 bufio.(*Reader).Peek(0xc0006e1f00, 0x1) bufio/bufio.go:144 +0x5d github.com/emersion/go-message/textproto.ReadHeader(0x0?) github.com/emersion/go-message@v0.15.0/textproto/header.go:525 +0x5f git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessageHeaders.func1(0xc00026e980) git.sr.ht/~rjarry/aerc/worker/imap/fetch.go:49 +0x1ab git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages.func1() git.sr.ht/~rjarry/aerc/worker/imap/fetch.go:222 +0xa2 created by git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages git.sr.ht/~rjarry/aerc/worker/imap/fetch.go:217 +0x10a Some IMAP messages only contain flag updates, ignore them. Reuse the same logic than handleFetchMessageBodyPart() and handleFechFullMessages(). Fixes: https://todo.sr.ht/~rjarry/aerc/111 Signed-off-by: Robin Jarry <robin@jarry.cc> Tested-by: normen <normenweb@mac.com>
* message-info: add explicit References fieldTim Culverhouse2022-11-091-0/+4
| | | | | | | | | | Add an explicit References field to message info. This is useful for storing information needed for threading without storing all of the header values, keeping system RAM usage lower. Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Tested-by: Inwit <inwit@sindominio.net> Acked-by: Robin Jarry <robin@jarry.cc>
* viewer: option to not mark message as seenKoni Marti2022-10-041-1/+4
| | | | | | | | | | | | | | | | | Add option to open a message in the message viewer without setting the seen flag. Enables the message viewer to be used as a preview pane without changing the message flags unintentionally. Before, the message viewer would set the seen flag by default. The IMAP backend will now always fetch the message body with the peek option enabled (same as we fetch the headers). An "auto-mark-read" option is added to the ui config which is set to true by default. If set the false, the seen flag is not set by the message viewer. Co-authored-by: "James Cook" <falsifian@falsifian.org> Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
* backends: send MessageInfoError on header fetching errorTim Culverhouse2022-09-251-4/+1
| | | | | | | | | | | | | | | | | | | | | When an error is encountered fetching a header, the backends respond with a type.Error worker message. On receipt of this message, the UI deletes all pending headers. The headers are all requested again as they remain on the screen, resulting in an infinite request loop - and an infinite logging loop. The user only ever sees the spinner unless they check the logs. A previous commit intended to fix this, however it introduced a regression where any message that was part of the fetch request would also be marked as erroneous. This commit is reverted with commit 2aad2fea7d36 ("msgstore: revert 9fdc7acf5b48"). Send an erroneous message info message from the backend when an error is encountered for a specific UID. Fixes: 01f80721e283 ("msgstore: post MessageInfo on erroneous fetch") Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Acked-by: Robin Jarry <robin@jarry.cc>
* charset: handle unknown charsets more user-friendlyKoni Marti2022-09-251-1/+4
| | | | | | | | | Do not throw an error when the charset is unknown; the message entity can still be read, but log the error instead. Reported-by: falsifian Signed-off-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
* lint: work nicely with wrapped errors (errorlint)Moritz Poldrack2022-08-041-2/+2
| | | | | | | | Error wrapping as introduced in Go 1.13 adds some additional logic to use for comparing errors and adding information to it. Signed-off-by: Moritz Poldrack <moritz@poldrack.dev> Acked-by: Robin Jarry <robin@jarry.cc>
* seqmap: refactor seqmap to use slice instead of mapTim Culverhouse2022-08-031-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The imap worker's seqmap is represented as a map of sequence number to UID. This presents a problem when expunging group of messages from the mailbox: each individual expunge decrements the sequence numbers by 1 (for every sequence number greater than the expunged). This requires a looping around the map to update the keys. The use of a map also requires that both the sequence number and the UID of a message be known in order to insert it into the map. This is only discovered by fetching individual message body parts (flags, headers, etc), leaving the seqmap to be empty until we have fetched information about each message. In certain instances (if a mailbox has recently been loaded), all information is loaded in memory and no new information is fetched - leaving the seqmap empty and the UI out of sync with the worker. Refactor the seqmap as a slice, so that any expunge automatically decrements the rest of the sequences. Use the results of FetchDirectoryContents or FetchDirectoryThreaded to initialize the seqmap with all discovered UIDs. Sort the UIDs in ascending order: IMAP specification requires that sequence numbers start at 1 increase in order of ascending UID. Add individual messages to the map if they come via a MessageUpdate and have a sequence number larger than our slice. Update seqmap tests with new logic. Reference: https://datatracker.ietf.org/doc/html/rfc3501#section-2.3.1.2 Fixes: https://todo.sr.ht/~rjarry/aerc/69 Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Acked-by: Robin Jarry <robin@jarry.cc>
* lint: apply new formatting rulesMoritz Poldrack2022-08-011-7/+8
| | | | | | | Run `make fmt`. Signed-off-by: Moritz Poldrack <git@moritz.sh> Acked-by: Robin Jarry <robin@jarry.cc>
* logging: use level-based logger functionsRobin Jarry2022-07-231-5/+5
| | | | | | | | | | | | Do not pass logger objects around anymore. Shuffle some messages to make them consistent with the new logging API. Avoid using %v when a more specific verb exists for the argument types. The loggers are completely disabled (i.e. Sprintf is not even called) by default. They are only enabled when redirecting stdout to a file. Signed-off-by: Robin Jarry <robin@jarry.cc> Acked-by: Moritz Poldrack <moritz@poldrack.dev>
* imap: fix error when server returns a message without body sectionRobin Jarry2022-07-231-0/+4
| | | | | | | | | | | | | | | | | | | | | | When opening unread emails from certain people (I won't name any names, sorry), an annoying error message is displayed on the status line: could not get section &imap.BodySectionName{BodyPartName: imap.BodyPartName{Specifier:"", Path:[]int(nil), Fields:[]string(nil), NotFields:false}, Peek:false, Partial:[]int(nil), value} This does not occur for already read messages. This issue is similar to the one that was fixed in commit 8ed95b0d2ad2 ("imap: avoid crash when replying to unread message"). This happens because the flags are updated in the callback that receives the message itself. It causes the flag update to arrive in the same channel/request. Ignore the messages that have an empty body (i.e. only containing flag updates). This is inherently racy but there seems no way to get rid of these extra messages. Signed-off-by: Robin Jarry <robin@jarry.cc> Acked-by: Koni Marti <koni.marti@gmail.com>
* imap: fix data race on seqMap arrayRobin Jarry2022-06-241-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | There are concurrent threads that are accessing and modifying IMAPWorker.seqMap (the mapping of sequence numbers to message UIDs). This can lead to crashes when trying to add and remove a message ID. panic: runtime error: index out of range [391] with length 390 goroutine 1834 [running]: git.sr.ht/~rjarry/aerc/logging.PanicHandler() logging/panic-logger.go:47 +0x6de panic({0xa41760, 0xc0019b3290}) /usr/lib/golang/src/runtime/panic.go:838 +0x207 git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages.func1() worker/imap/fetch.go:214 +0x185 created by git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages worker/imap/fetch.go:209 +0x12b Use a map which makes more sense than a simple array for random access operations. Also, it allows better typing for the key values. Protect the map with a mutex. Add internal API to access the map. Add basic unit tests to ensure that concurrent access works. Fixes: https://todo.sr.ht/~rjarry/aerc/49 Signed-off-by: Robin Jarry <robin@jarry.cc> Acked-by: Moritz Poldrack <moritz@poldrack.dev>
* imap: add option to cache headersTim Culverhouse2022-06-221-10/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add option to cache headers for imap accounts. Cache db is located at $XDG_CACHE_DIR/aerc/{account name}. The cache is cleaned of stale entries when aerc is first opened. Two new account level configuration options are introduced: * cache-headers (Default: false) * cache-max-age (Default: 30 days (720 hours)) The change in worker/imap/open.go is to set the selected directory. This is required to access the UIDVALIDITY field, which is used in combination with the message ID to form the key for use in the cache db. The key structure is: "header.{UIDVALIDITY}.{UID}" Where reasonable, cache does not stop aerc from running. In general, if there is an error in the cache, aerc should continue working as usual. Errors are either displayed to the user or logged. All messages are stored without flags, and when retrieved have the flags set to SEEN. This is to prevent UI flashes. A new method to FetchMessageFlags is introduced to update flags of cached headers. This is done asynchronously, and the user will see their messages appear and then any flags updated. The message will initially show as SEEN, but will update to unread. I considered updating the cache with the last-known flag state, however it seems prudent to spare the R/W cycle and assume that - eventually - all messages will end up read, and if it isn't the update will occur rather quickly. Note that leveldb puts a lock on the database, preventing multiple instances of aerc from accessing the cache at the same time. Much of this work is based on previous efforts by Vladimír Magyar. Implements: https://todo.sr.ht/~rjarry/aerc/2 Thanks: Vladimír Magyar <vladimir@mgyar.me> Signed-off-by: Tim Culverhouse <tim@timculverhouse.com> Tested-by: inwit <inwit@sindominio.net> Reviewed-by: Koni Marti <koni.marti@gmail.com> Acked-by: Robin Jarry <robin@jarry.cc>
* imap: avoid crash when replying to unread messageRobin Jarry2022-04-281-2/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When running `:reply -q` on an unread message, aerc crashes after opening the editor: panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x5d1019] goroutine 63 [running]: bufio.(*Reader).fill(0xc000086ef8) /usr/lib/golang/src/bufio/bufio.go:106 +0xd9 bufio.(*Reader).Peek(0xc00020bef8, 0x1) /usr/lib/golang/src/bufio/bufio.go:144 +0x5d github.com/emersion/go-message/textproto.ReadHeader(0xc00004a700?) emersion/go-message@v0.15.0/textproto/header.go:525 +0x5f git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessageBodyPart.func1(0xc00056e280) worker/imap/fetch.go:99 +0x1ab git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages.func1() worker/imap/fetch.go:178 +0xd7 created by git.sr.ht/~rjarry/aerc/worker/imap.(*IMAPWorker).handleFetchMessages worker/imap/fetch.go:172 +0x12b This happens because the flags are updated in the callback that receives the message itself. It causes the flag update to arrive in the same channel/request. Ignore the messages that have an empty body (i.e. only containing flag updates). This is inherently racy but there seems no way to get rid of these extra messages. Signed-off-by: Robin Jarry <robin@jarry.cc> Tested-by: Connor Kuehl <cipkuehl@gmail.com>
* logging: added a log on panicMoritz Poldrack2022-03-231-0/+3
| | | | | | | | | | | | | Since panics still regularly "destroy" the terminal, it is hard to get a stack trace for panics you do not anticipate. This commit adds a panic handler that automatically creates a logfile inside the current working directory. It has to be added to every goroutine that is started and will repair the terminal on a panic. Signed-off-by: Moritz Poldrack <git@moritz.sh> Acked-by: Robin Jarry <robin@jarry.cc>
* go vet: composite literal uses unkeyed fieldsMoritz Poldrack2022-03-181-3/+3
| | | | | | | | This commit fixes all occurrences of the abovementioned lint-error in the codebase. Signed-off-by: Moritz Poldrack <git@moritz.sh> Acked-by: Robin Jarry <robin@jarry.cc>
* imap: do not abort on malformed messagesRobin Jarry2022-02-061-1/+7
| | | | | | | | | | | | | | Some messages contain invalid headers, textproto.ReadHeader fails with an error: malformed MIME header key: From nobody Fri Jan 14 19 And all other messages of the selected folder are ignored. Return an explicit error message to the UI and continue processing other messages. This is consistent with what the maildir worker does. Signed-off-by: Robin Jarry <robin@jarry.cc>
* go.mod: change base git urlRobin Jarry2021-11-051-2/+2
| | | | | | | I'm not sure what are the implications but it seems required. Link: https://github.com/golang/go/issues/20883 Signed-off-by: Robin Jarry <robin@jarry.cc>
* imap: Remove FetchMessageBodyPart.{Encoding,Charset}Reto Brunner2020-05-161-106/+107
| | | | Fixes https://todo.sr.ht/~sircmpwn/aerc2/352 exactly as suggested by emersion.
* imap/fetch.go: report error when fetching headers failsBen Fiedler2020-04-241-3/+4
| | | | | Correct me if I'm wrong, but shouldn't this raise an error when it fails?
* Ignore missing message part in getDecodedPartAmir Yalon2020-03-091-1/+1
| | | | The code that calls this function handles nil without issues.
* Mark sent messages as "seen" in maildirGalen Abell2020-03-031-3/+3
| | | | | | | - Add maildir flags to complement a messages imap flags - Set the "seen" flag on sent messages when using the maildir backend - Cleanup AppendMessage interface to use models.Flag for both IMAP and maildir
* worker/imap: don't decode in FetchFullMessage.Reto Brunner2020-01-181-27/+2
| | | | | Doing that breaks `git am` as it expected the encoded variant. Same is probably true for any sort of signature validation (gpg / dkim)
* imap: decode reader prior to returning themReto Brunner2020-01-051-10/+90
|
* Factor IMAP-specific structs out of UI modelsBen Burwell2019-07-081-5/+5
| | | | | | | Before, we were using several IMAP-specific concepts to represent information being displayed in the UI. Factor these structures out of the IMAP package to make it easier for other backends to provide the required information.
* Use []uint32 instead of imap.SeqSetBen Burwell2019-07-081-8/+6
| | | | | | | | A sequence-set is an IMAP-specific implementation detail. Throughout the UI, aerc simply operates using lists of opaque identifiers. In order to loosen the coupling between the UI and IMAP in particular, replace most usages of imap.SeqSet with []uint32, leaving the translation to a SeqSet to the IMAP backend as needed.
* Factor UI models out of the worker message packageBen Burwell2019-07-081-15/+26
| | | | | | | | Before, the information needed to display different parts of the UI was tightly coupled to the specific messages being sent back and forth to the backend worker. Separating out a models package allows us to be more specific about exactly what a backend is able to and required to provide for the UI.
* Update our message flags when server updates themDrew DeVault2019-06-091-2/+22
|
* Fetch headers w/peek to leave emails unreadYash Srivastav2019-06-081-0/+1
|
* implements ability to view headers in message viewYash Srivastav2019-06-071-1/+20
|
* Don't send Done until finished fetching messagesDrew DeVault2019-06-021-0/+3
|
* Flesh out multipart switcherDrew DeVault2019-05-201-1/+1
|
* s/aerc2/aerc/gDrew DeVault2019-05-171-1/+1
|
* worker/imap: use the IMAP connection from a single goroutineSimon Ser2019-04-291-41/+37
| | | | | | | | | | | | | Unfortunately, the IMAP protocol hasn't been designed to be used from multiple goroutines at the same time. For instance, if you fetch twice the same message from two different goroutines, it's not possible to tell whether the response is for one receiver or the other. For this reason, go-imap clients aren't safe to use from multiple goroutines. This commit changes the IMAP workers to be synchronous again (a command is executed only after the previous one has completed). To use IMAP from different threads, popular clients (e.g. Thunderbird) typically open multiple connections.
* Make message viewer real, part twoDrew DeVault2019-03-311-11/+14
|
* Rename FetchMessageBodies to FetchFullMessagesDrew DeVault2019-03-311-10/+10
|
* Make the message viewer real, part oneDrew DeVault2019-03-311-6/+26
|
* Don't parse mail in worker; send a reader insteadDrew DeVault2019-03-311-10/+2
|
* Pull BodyStructure up from IMAP workerDrew DeVault2019-03-311-4/+6
|
* Add body fetching support codeDrew DeVault2019-03-291-15/+46
|
generated by cgit (git 2.34.1) at 2024-10-21 01:28:53 +0000