Unbelievable

[rowyn]

Latest on the Sidekick, direct from T-Mobile.

Allow me to quote the most relevant bit:

Based on Microsoft/Danger's latest recovery assessment of their systems, we must now inform you that personal information stored on your device - such as contacts, calendar entries, to-do lists or photos - that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger.

...

I am still trying to wrap my head around this.

I understand how a company can fail to have a recent backup. That happens a lot. You do a nightly backup or something and you lose the last 12 hours of data since the last save. Or you do a nightly backup and when you try to restore, you discover the backup is corrupted and you go to an older one. OpenDiary had a hacker attack some years ago to which they lost not only the live environment but three or four months' worth of backups. But they did, eventually, go back far enough to find a backup that worked. They were able to restore. Three months out of date, yes, and that was catastrophic. But they had something.

Microsoft/Danger's got nothing?

Seriously? Nothing?

I can't even believe it. Literally, I am struggling to believe that they mean what they wrote. makovette linked me to a third party site that reproduced the letter, and I had to go to T-Mobile's page myself to verify that was really what they were saying. Because I can't believe it.

O_O

Five or so months ago, I switched my primary email to GMail. My main reason: I figured Google was probably better at backing up my data than I was.

This makes me rethink the wisdom of cloud computing. Yes, Danger was a small company and they'd lost most of their talent since the acquisition. But they were owned and managed by Microsoft. Microsoft, people! However much people may mock them, this is not a company widely known for gross negligence and incompetence. If Microsoft can't manage some level of data integrity, how can I trust anyone to?

I told this to Lut just now and he didn't even believe me. I had to show him the webpage. How is that even possible? How can you have nothing?

I was not one of the unfortunate ones whose Sidekick lost all memory during the outage. That happens sometimes; the memory gets wiped by a glitch, or a total loss of power, and your SK has to re-sync with the server to restore all data. It's happened to me a few times, but re-syncing doesn't take long and I never lost anything. Except that if it had happened to me in the last week, I would have. If it happens to me in the next few days, I'll lose everything.

But right now, my SK still has everything stored locally. Presumably, at some point they will finish restoring the server, sync, and back up the local data. One might guess that they will learn from this disaster and have better data recovery processes in the future, but I can't say that I really trust them, or anyone else, any more. So I just finished emailing each of my 100 contacts, one at a time, to my gmail account. So I've got a backup now.

...

Seriously? Nothing?

...

I think I'll go find a tool to back up my livejournals now.

Comments

[makovette.livejournal.com]

The number of people being fired on Monday from the responsible group will be Large.

Does the customers no good of course, but at least heads will roll.

CYa!
Mako

[terrycloth.livejournal.com]

I've heard of this sort of thing happening before, although not in a customer-facing context. Sometimes there's a fatal flaw in your backup procedure that isn't apparent until you first try to go and restore from the backups.

Like, tape drives that slowly shifted alignment as time went on, meaning that old tapes couldn't be read.

Or a script that moves backups around to make room for the most recent one, that through some programming flaw deleted the backups instead of moving them. One story I head of was one that through some y2k-like change in the environment suddenly shifted from 'working properly' to 'devouring all backups and taking down the original server too'.

So, it's not necessarily just 'we decided we didn't want to bother making backups'. Altough that's a possibility too. People *are* lazy.

[jurann.livejournal.com]

Sounds like the basis for a class action lawsuit! =D

[xthread.livejournal.com]

However much people may mock them, this is not a company widely known for gross negligence and incompetence.

Ahahahahahaha... <collapses in giggles>

Thank you, I needed a funny this evening.

[terrycloth.livejournal.com]

If it goes really well, you might get a coupon for a free month of service!

[rowyn]

I thought that standard disaster-recovery procedure -- y'know, like what my bank uses for its customers -- go something like this:

1) Regular backups made nightly.
2) Samples of old backups are restored periodically (maybe only weekly or monthly) to ensure that the backup process is working.
3) Backups are moved offsite regularly to ensure that they can't get corrupted.

If a company is doing all three of those, it seems hard to imagine a glitch that would take out everything.

OTOH, I work for a bank, and we really, seriously, have to not dick around with this. We not only have a disaster recovery process that assumes our main IT location was wiped off the face of the Earth, but we have full-scale annual tests to make sure it all works. By which I mean "dozens of workers come in on Saturday afternoon and pretend to do all their regular work using the backup system at a different location using different protocols".

And I guess I can see how Microsoft/Danger might not have had the luxury of doing a real test, because doing a real test generally means taking your main system offline and using your backup, and that is generally going to lead to a service disruption. T-Mobile may have refused them permission to do that kind of testing. And maybe whatever test they were able to do weren't enough to uncover the gaping holes in their backup process that made it completely useless.

... Hmm. I think that scenario makes sense.

It also makes me feel rather leery of any service (like gmail! Or Livejournal!) that doesn't go down for scheduled maintenance.

I don't know. It's just so totally bizarre to me that it's making me paranoid about everything. I made a backup of both of my journals tonight. I wish I knew a way to archive my GMail.

[rowyn]

Already got one. :P

[rowyn]

I do almost want to sue them for breach of contract, just so that I can switch to another service now instead of waiting for my contract to be up.

[rowyn]

What I really want is to hear the whole story, and find out why they didn't have a working backup. I want to know all those technical details they left out of their summation.

[terrycloth.livejournal.com]

I've always been really leery about trusting data storage on the web. Not because I thought that a company would accidentally delete everything, but because you never know when a company is going to suddenly stop offering its services.

Same deal as DRM, basically.

[argonel.livejournal.com]

You should be able to connect to gmail using a normal e-mail client over IMAP and download a complete copy of all your mail. Check the following link for more information and instructions.

http://mail.google.com/support/bin/topic.py?hl=en&topic=12806

[rowyn]

Oooh, thanks for the tip!

[rowyn]

Yeah, there's that, too. Even so, most companies don't close up over night and vanish, so there's usually time to try to rescue your stuff before they fold.

[sebkha.livejournal.com]

Yeah, my normal mode of GMail use is to download everything into my local email client and delete the online copies. I'm actually motivated for the opposite reason though -- I'd prefer Google not have copies of all my mail. You could certainly leave a copy of your mail online if remote backups were your priority, though.

[rowyn]

I admit it doesn't make a lot of sense to me that I trust Google with all my email when I don't really trust my government with the information on my tax return. :/ Possibly the difference is that Google is actually doing me some good by serving as a remote backup, and the government isn't really doing me any good with most of the info it's collecting on me. >:)

Even so, I'm not entirely unworried about Google. They have a lousy track record when it comes to customer privacy vs government request. :/ Do they really destroy your email when you trash it, beyond their ability to recover?

[sebkha.livejournal.com]

They claim We will make reasonable efforts to remove deleted information from our systems as quickly as is practical. I suspect that have better motivation to be truthful than to be deceptive.

[makovette.livejournal.com]

Barring those details coming out as part of a class action lawsuit that goes to court instead of settlement, we're unlikely to ever get the complete story.

CYa!
Mako

[sythyry.livejournal.com]

*boggle* That's ridiculous.

[foomf.livejournal.com]

Oh, really.

Up until YESTERDAY I had not lost data. YESTERDAY they pushed this broken patch to my phone, and it lost some pretty much irreplaceable stuff.

Yes, it is the 15th, yes, it was pushed WELL AFTER they knew it was broken, and yes, I had followed every bit of advice they gave EXCEPT for going through the step of manually going contact-by-contact through my phone to save a partial record to my SIM card.

And YESTERDAY the customer service agent told me that "They hope to be able to recover it soon."

I look forward to participating in the class action lawsuit if they don't.


AND, looking at the page in question, which I quote here:

Updated: 10/15/2009 1:00 AM PDT
Microsoft Confirms Data Recovery for Sidekick Users

Data Restoration to Begin as Soon as Possible for Affected Customers

Dear T-Mobile Sidekick customers,

On behalf of Microsoft, I want to apologize for the recent problems with the Sidekick service and give you an update on the steps we have taken to resolve these problems.

We are pleased to report that we have recovered most, if not all, customer data for those Sidekick customers whose data was affected by the recent outage. We plan to begin restoring users’ personal data as soon as possible, starting with personal contacts, after we have validated the data and our restoration plan. We will then continue to work around the clock to restore data to all affected users, including calendar, notes, tasks, photographs and high scores, as quickly as possible.

We now believe that data loss affected a minority of Sidekick users. If your Sidekick account was among those affected, please continue to log into these forums for the latest updates about when data restoration will begin, and any steps you may need to take. We will work with T-Mobile to post the next update on data restoration timing no later than Saturday.

We have determined that the outage was caused by a system failure that created data loss in the core database and the back-up. We rebuilt the system component by component, recovering data along the way. This careful process has taken a significant amount of time, but was necessary to preserve the integrity of the data.

We will continue working closely with T-Mobile to restore user data as quickly as possible. We are eager to deliver the level of reliable service that our incredibly loyal customers have become accustomed to, and we are taking immediate steps to help ensure this does not happen again. Specifically, we have made changes to improve the overall stability of the Sidekick Service and initiated a more resilient backup process to ensure that the integrity of our database backups is maintained.

Once again, we apologize for this situation and the inconvenience that it has created. Please know that we are working all-out to resolve this situation and restore the reliability of the service.


Sincerely,
Roz Ho
Corporate Vice President
Premium Mobile Experiences, Microsoft Corporation


Uh. WHY then did it flush my data YESTERDAY?
I will probably stick with T-Mobile, as 10+ years of customer loyalty seems to be worth something to them. I will, however, be changing my phone. And I expect them to eat the cost of swapping it.

[rowyn]

Wow, that sucks. :( I haven't lost any data on my handset (yet). I have gone through and emailed almost everything on my SK to my gmail account, including 800+ saved emails and 100+ contacts. I still have notes I need to email to myself. And there's no good way I know of to export my calendar. :/

I was planning on migrating off the SK anyway, since it was clear even before this that the line wasn't being supported any more. The 2009 SKs are barely different from the 2008 ones. It's a pity, really. It was a wonderful device in its time. It's not quite like watching the Amiga fail, but that what it reminds me of.

[the-vulture.livejournal.com]

Crikey! Makes me glad I'm a Crackberry addict; I can take care of my own backups (saved on two different systems), so I don't have to worry about such strangeness.

[the-vulture.livejournal.com]

That might be worth a shot.