Unbelievable

[rowyn]

Latest on the Sidekick, direct from T-Mobile.

Allow me to quote the most relevant bit:

Based on Microsoft/Danger's latest recovery assessment of their systems, we must now inform you that personal information stored on your device - such as contacts, calendar entries, to-do lists or photos - that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger.

...

I am still trying to wrap my head around this.

I understand how a company can fail to have a recent backup. That happens a lot. You do a nightly backup or something and you lose the last 12 hours of data since the last save. Or you do a nightly backup and when you try to restore, you discover the backup is corrupted and you go to an older one. OpenDiary had a hacker attack some years ago to which they lost not only the live environment but three or four months' worth of backups. But they did, eventually, go back far enough to find a backup that worked. They were able to restore. Three months out of date, yes, and that was catastrophic. But they had something.

Microsoft/Danger's got nothing?

Seriously? Nothing?

I can't even believe it. Literally, I am struggling to believe that they mean what they wrote. makovette linked me to a third party site that reproduced the letter, and I had to go to T-Mobile's page myself to verify that was really what they were saying. Because I can't believe it.

O_O

Five or so months ago, I switched my primary email to GMail. My main reason: I figured Google was probably better at backing up my data than I was.

This makes me rethink the wisdom of cloud computing. Yes, Danger was a small company and they'd lost most of their talent since the acquisition. But they were owned and managed by Microsoft. Microsoft, people! However much people may mock them, this is not a company widely known for gross negligence and incompetence. If Microsoft can't manage some level of data integrity, how can I trust anyone to?

I told this to Lut just now and he didn't even believe me. I had to show him the webpage. How is that even possible? How can you have nothing?

I was not one of the unfortunate ones whose Sidekick lost all memory during the outage. That happens sometimes; the memory gets wiped by a glitch, or a total loss of power, and your SK has to re-sync with the server to restore all data. It's happened to me a few times, but re-syncing doesn't take long and I never lost anything. Except that if it had happened to me in the last week, I would have. If it happens to me in the next few days, I'll lose everything.

But right now, my SK still has everything stored locally. Presumably, at some point they will finish restoring the server, sync, and back up the local data. One might guess that they will learn from this disaster and have better data recovery processes in the future, but I can't say that I really trust them, or anyone else, any more. So I just finished emailing each of my 100 contacts, one at a time, to my gmail account. So I've got a backup now.

...

Seriously? Nothing?

...

I think I'll go find a tool to back up my livejournals now.

Comments

[terrycloth.livejournal.com]

I've heard of this sort of thing happening before, although not in a customer-facing context. Sometimes there's a fatal flaw in your backup procedure that isn't apparent until you first try to go and restore from the backups.

Like, tape drives that slowly shifted alignment as time went on, meaning that old tapes couldn't be read.

Or a script that moves backups around to make room for the most recent one, that through some programming flaw deleted the backups instead of moving them. One story I head of was one that through some y2k-like change in the environment suddenly shifted from 'working properly' to 'devouring all backups and taking down the original server too'.

So, it's not necessarily just 'we decided we didn't want to bother making backups'. Altough that's a possibility too. People *are* lazy.

[rowyn]

I thought that standard disaster-recovery procedure -- y'know, like what my bank uses for its customers -- go something like this:

1) Regular backups made nightly.
2) Samples of old backups are restored periodically (maybe only weekly or monthly) to ensure that the backup process is working.
3) Backups are moved offsite regularly to ensure that they can't get corrupted.

If a company is doing all three of those, it seems hard to imagine a glitch that would take out everything.

OTOH, I work for a bank, and we really, seriously, have to not dick around with this. We not only have a disaster recovery process that assumes our main IT location was wiped off the face of the Earth, but we have full-scale annual tests to make sure it all works. By which I mean "dozens of workers come in on Saturday afternoon and pretend to do all their regular work using the backup system at a different location using different protocols".

And I guess I can see how Microsoft/Danger might not have had the luxury of doing a real test, because doing a real test generally means taking your main system offline and using your backup, and that is generally going to lead to a service disruption. T-Mobile may have refused them permission to do that kind of testing. And maybe whatever test they were able to do weren't enough to uncover the gaping holes in their backup process that made it completely useless.

... Hmm. I think that scenario makes sense.

It also makes me feel rather leery of any service (like gmail! Or Livejournal!) that doesn't go down for scheduled maintenance.

I don't know. It's just so totally bizarre to me that it's making me paranoid about everything. I made a backup of both of my journals tonight. I wish I knew a way to archive my GMail.

[terrycloth.livejournal.com]

I've always been really leery about trusting data storage on the web. Not because I thought that a company would accidentally delete everything, but because you never know when a company is going to suddenly stop offering its services.

Same deal as DRM, basically.

[rowyn]

Yeah, there's that, too. Even so, most companies don't close up over night and vanish, so there's usually time to try to rescue your stuff before they fold.

[argonel.livejournal.com]

You should be able to connect to gmail using a normal e-mail client over IMAP and download a complete copy of all your mail. Check the following link for more information and instructions.

http://mail.google.com/support/bin/topic.py?hl=en&topic=12806

[rowyn]

Oooh, thanks for the tip!

[sebkha.livejournal.com]

Yeah, my normal mode of GMail use is to download everything into my local email client and delete the online copies. I'm actually motivated for the opposite reason though -- I'd prefer Google not have copies of all my mail. You could certainly leave a copy of your mail online if remote backups were your priority, though.

[rowyn]

I admit it doesn't make a lot of sense to me that I trust Google with all my email when I don't really trust my government with the information on my tax return. :/ Possibly the difference is that Google is actually doing me some good by serving as a remote backup, and the government isn't really doing me any good with most of the info it's collecting on me. >:)

Even so, I'm not entirely unworried about Google. They have a lousy track record when it comes to customer privacy vs government request. :/ Do they really destroy your email when you trash it, beyond their ability to recover?

[sebkha.livejournal.com]

They claim We will make reasonable efforts to remove deleted information from our systems as quickly as is practical. I suspect that have better motivation to be truthful than to be deceptive.