Dissecting this week's Evernote outage & did offline mode work
It’s been another exciting week in Evernote world!
Last week we were still talking about the new Ai Powered Search and experimenting with it, this week we’re dealing with a long outage! From a high to a low.
Evernote have published a pretty extensive blog post on what went wrong and I thought I’d quickly share my experiences of Wednesday.
I started work on Wednesday morning around 8am (UK time) and things were working fine. I checked off a few tasks, made some notes for a 3pm meeting and started working on a big client project.
This is a coding project and I have a huge amount of documentation stored inside Evernote so I know what I’m building. I’m inside Evernote all the time when working like this.
Around 9.30am I noticed the grey bar appear.
I checked WiFi and just put it down as a glitch but the bar didn't move. After about half an hour I checked the Evernote forums and found others were having the same issue.
I saw the grey bar for most of Wednesday. I think it was around 3 or 4pm (UK time) when things got back to normal.
So what happened?
A quick bit of history first. When Bending Spoons bought Evernote back in January, they inherited some pretty old code. The fundamentals of Evernote run using a very large single Java programme that looks after much of the functionality. This is difficult to work with and the new team are currently re-writing the entire back-end in a more modern way. Evernote wrote about it here and I’ve written about it here.
So, from what I know, on Wednesday the plan was to migrate a large database that was responsible for managing logins, new registrations and information related to the user profile to a new system. This happened.
It was when they started switching stuff back on and over to the new system that issues started to arise.
From the blog post:
With the migration complete, our servers were overwhelmed by the number of inbound requests as the clients attempted to reconnect
The Monolith was unable to scale to meet the higher demand.
Evernote then decided to:
temporarily block client requests and allow the servers to restart, which led to the outage.
The situation was that anyone already logged in was OK but working offline for some of the day. Logging in was not fully restored for quite a while as Evernote were rate-limiting logins to protect the systems.
The logging in and authorisation issue was affecting syncing, the web clipper and other connected services.
This was fully resolved on Thursday morning and everything should be working OK now.
Better comms
I’ve been in a similar situation myself where a large system goes down and everyone is running around like a SpongeBob SquarePants meme trying to work out what’s happening.
For at least the first hour there is so much information coming at you that it’s difficult to work out exactly what the problem is.
This said I think Evernote will be communicating better in the future. Just a quick update to socials saying there’s a problem and everyone’s working on it would help.
Once they knew what was happening, they did communicate on the user forums but more social posts would’ve been nice.
How did offline mode work?
Pretty well. I spent most of the day with the grey bar showing and to be honest I carried on as usual.
I was desktop only on Wednesday and managed to get access to all my notes, update notes, create new ones and just carry on as usual.
Once everything was back up and running all my changes were synced back up to the cloud.
One thing I did notice was that sometimes the app would be a little slow. I think this was because it was trying to reconnect with Evernote and wasn’t sure if it was meant to be working offline or online. I sometimes get this on my phone when I have a bad connection.
I’d love to see an Online/Offline Toggle for all the Evernote apps so I can force offline when I have a bad connection. Could be in a coffee shop, on the train or in Wales!
Even with the niggles I got my work done in offline mode.
I was kind of glad I had a note taking app that worked offline and not cloud only.
How did you get on?
Appearing on Vlad’s YouTube channel!
A big thank you to Vlad Campos for inviting me onto his YouTube channel to chat about Evernote’s Ai Powered Search. I loved the chat even though I was in the middle of a COVID fever dream! I hope I made sense.
Weekly Reading Recommendations
Something new for each of the newsletters.
I read a lot of Evernote and productivity content so I thought I'd share some of what I’m reading each week.
The people going 'monk mode' to limit social media use - BBC
The Future of Evernote. Interview with Federico on the Keep Productive YouTube channel.
Use the Rule of 3 to Improve Your Productivity - Paper Doll Blog
That’s it.
Have a great weekend
All the best
Jon
@Jon - Thanks for your continued Evernote posts. Even if they don't talk about new features, they arev useful and informative. I seem to have been "lucky" regarding the October 4th outage - while I was aware that there were problems when I first logged in just before 9am that morning, I didn't need great use of EN until later in the day, and then things seemed like normal.
Over the last 3 weeks or so, it's felt that EN has performed well - good response time, syncing correctly, etc. So I'm staying with EN as my "second brain".
Will need to experiment with the AI search - my tendency is to focus on keywords rather than clever enquiries; I do think it will reduce my use of the filters option, but we'll see.
LOL "Better comms" 🤣 There wasn't any comms whatsoever so it would be difficult to have worse comms. I've been an Evernote user for a long time and I cannot remember a period of outage such as this so they are very rare and I personally don't mind so long as they are open and giving regular updates. This week's lack of updates was unacceptable in my opinion. They have to do better in this area going forward.