Re: [Evolution] Migrating from Thunderbird via import without duplicates

[Pete Biggs <pete biggs org uk>]

On Thu, 2020-10-29 at 01:09 +0900, Bo Buckley wrote:
> Hi Pete (sorry for the missed reply-all last time),

Yeah, please try and do reply-to-list not reply-all!

> Thank you for your reply. [And thank you to everyone else for your insight as well!]
>
> > A serious question: if you are using IMAP, why do you need to "sync"
> > all your mail?  
>
> Because there is no backup on the server that I can restore from and
> if I press select all -> delete in Evolution all my mail goes to the
> ether without so much as a "are you sure?" dialog. 

There should be a confirmatory popup on deletion - at least the first
time, you can tick "don't ask me again" though.

> This is obviously a worst case scenario, but at literally three (two
> if you count ctrl+a as one ;) ) quick keyboard clicks and no
> confirmation, its a real worry to me.

Yes, but mail is never deleted that easily. It will always, I think, be
put into a Trash first, then you have to explicitly expunge it to
permanently get rid of it.

>  There are various less severe scenarios where I'd like to have a
> backup of all my mail somewhere. Trusting the IMAP server to never
> fail is quite risky in my opinion. Do you have a better way to keep a
> backup than what I am suggesting?

I use ISP's and mail providers that make backups regularly. If yours
doesn't, then I might hazard a suggestion that you aren't using the
correct provider for your risk perception.

> Also I like to keep a local copy of all mail as I am offline quite
> often and need to have all work related email to reference various
> things from years past. 

The local cache will provide that. But it's a cache and considered
expendable.  You also have to consider coherency between your local
copies and the IMAP server - what happens if you modify the local copy
when your offline - how do the changes get passed back to the server
and so on. It's not a trivial process.

> Another reason is the speed in which I can search for old mail. For
> example if I have a client I want to contact again, but don't
> remember their name, I can look up the project number and the mail
> thread will be right there within a second or two. Google's online
> interface is quick enough to do this at about the same speed, but as
> for the mail on my justhost.com account, that could take several
> minutes before I get a full list of results. 

Again, that's an issue with your mail provider - I have all my mail
remote, albeit mostly on my own servers, and the mail is indexed when
it comes in and searching is stupidly fast - I can get search results
back from a 200,000 message folder in seconds.

> An even more recent example was when I started this thread. I added
> my foo bar com account to Evolution, and copy pasted my local copies
> to the Inbox. I ended up with a bunch of duplicates and Evolution's
> "remove duplicates" was not able to remove them all for some reason.
> Now I have a total mess of an Inbox. The quickest way to fix this was
> to just delete everything and redump the local backup copies. How
> would you fix any large scale issue like this without a backup?  

That's the issue of playing around with Evolution's internal files. If
your mail is remote, only, then you won't get into that sort of
trouble. If the duplicates were only local (and hadn't got copied back
to the IMAP server), then you just delete the account and start again.
If things had got back to the server, then ask your ISP to restore from
a backup.

> > Evolution keeps a copy of any mail you read as a local
> > cache, there's no permanent store of the mail, it's not a duplicate of
> > your IMAP account.
>
> Doesn't that depend on whether or not you enable the "Synchronize
> remote mail locally in all folders" option in Account Editor ->
> Receiving Options? 

AFAIK that's still just a cache - but it just pre-cache's it rather
than only caching messages as you read them. Others will probably
correct me.

> What I see locally is a mirror of what is on the mail server if I ssh
> into it after everything has synced. Am I overlooking something?

Where do you see it locally? 

> > If you need a local store of *all* your mail, then you might like to
> > look at something like Offline IMAP - this will drag down all your mail
> > and keep it locally, then you point Evolution at that store rather than
> > the actual IMAP account.  You may be better looking at that to pre-
> > populate with your extant local copy.
>
> Thats an interesting idea, but how would I handle outgoing mail at
> that point?

How do you handle outgoing mail when you are offline now? It will be
the same - Offline IMAP basically makes your local copy the definitive
one and handles the syncing and coherency with the server. Everything
else is the same.

> Ideally I'd like the following setup:
>
> 1.) Something like Offline IMAP or Evolution's "Synchronize remote
> mail locally in all folders" to sync to a local maildir
> 2.) Have an incremental backup of this maildir (e.g rsnapshot on
> cron) so that I have an actual backup and not just a mirror
> 3.) Send and receive emails through Evolution 
>
> I'm still not exactly sure how step three would look exactly. Keep it
> as a cache only imap account? Then pull a copy of the sent message
> from the mail server via step 1? 

No, the local copy is the definitive one and you use it like any other
account - Offline IMAP ensures that the server version is kept in sync
with your local one: if Evolution puts a copy of a message in the Sent
folder locally, it will also appear in the Sent folder on the server.
If your mail server handles putting things in the Sent folder, then
that message will be synced to the local Sent folder.

> Maybe a better question to the entire community would be how do you
> keep regularly scheduled backups of your mail? For self-hosted
> servers its a no-brainer, but for gmail accounts or other cloud
> hosted accounts, I'm really at a loss as to why I cannot find a
> standard way of doing this.

Well gmail is "backed up" - or rather nothing is ever deleted or
removed and there are multiple copies in data centres around the world.
All other mail providers should also back up things for disaster
recovery and anything that uses AWS or O365 will be in a similar
position to gmail. 

P.