SQL Server Crash
I know this Blog hasn't been around very long, it's only been about 6 weeks. But just as things were starting to look good and traffic was starting to grow, my SQL Server crashed. My objective here is to draw attention to simple ways we can all make a contribution to ecology and maybe draw some attention to other things I believe in.
You gotta understand this Blog isn't sitting on a laptop in my closet; it's also not sitting on some huge autonomous server farm. These are the servers I have configured for my clients. We're in a high quality, high speed Green Friendly datacenter right next to One Wilshire, he peering point for the entire West Coast. The SQL server is Microsoft 2005 Enterprise SQL server, obvious overkill but being a Microsoft Certified Partner, I got a deal.
The hardware is Dell running RAID 1 (which means I have 2 hard drives mirrored so they have exact copies of the same data on two drives). The odds of that breaking... 1 in 400 and it broke. I know, you other tech geeks (and I say that with affection) are thinking if I had RAID 5 it might not have been a problem, the odds of a RAID 5 failure 1 in 10,000 but taking this tangent just a bit further, even that may be a problem. As hard drives get bigger, the reliability of RAID 5 decreases.
The Big Picture
You know there has to be a point and here it is... regardless of how careful I was, my SQL server failed. This is a crisis because I'm not the only one on that SQL Server. The Irvine Public Schools Foundation runs their After School program for the entire district there. The Southern California Veterinary Medical Association manages all of their members on this server. This for me and my company was a crisis.
Immediate and effective action, there is no time to blame, the server failed Tuesday June 1st. I found the problem (2 failed hard drives). Once it was clear I couldn't get the data myself, I reached out to Data Recovery experts. I took the best drive to Al at Cyber Electronics in Orange County. Al recovered all of my data and the price was reasonable.
In the mean time, I had to consider the fastest way to get back online. I have a virtual server in the datacenter so I created an instance and built out a backup SQL Server there. Once I received the data, I was able to meticulously move the databases over one at a time, testing each one until they were all transferred and all were working Saturday June 5th, about 18 hours after I got the data back.
The point is this problem needed fixed and it needed fixed well. My clients weren't happy. But, mistakes like this happen and it's better to show direction and follow through than languish in possibilities.
Now that the crisis is contained, we get to go about making sure it doesn't happen again...
- We're rebuilding the Dell server. It's still good hardware.
- We will put monitoring in place that emails us if problems are starting (like a marginal hard drive).
- Once the new hardware is live, The Virtual server will become a mirror and kept as a warm standby (from a green point of view, this server is running anyway, as long as I'm not loading it, the impact is near zero).
- We back up the Virtual servers already, recovery from this is quick and simple, giving us 4 levels of redundancy so if there is a complete hardware failure of the first server (2 mirrored drives) and a complete hardware failure of the second server (RAID 5 with 4 drives), we can simply build new hardware and reboot the Virtual server files from the backup.
BP Oil Spill
Obviously, the quicker BP Oil gets their well online, the faster it starts making money. I heard it on KFI 640 Talk Radio last night and can't confirm it yet there was a safe way to finish up the well and a cheap way to finish up. An unnamed person at BP Oil insisted on doing it the cheap way. It's also my belief that BP Oil is more focused on stopping the oil from getting out than they are on stopping the oil that got out from hurting the environment.
I mentioned in an earlier Blog that they were releasing 5000 - 60,000 barrels of oil into the gulf. We know the "Cap" they put on is capturing 10,000 Barrels and that's possibly about 50% (BP says that's nearly all of it, I could be wrong but this video seems to indicate otherwise)
There are two problems to deal with, first, block the "Gusher" so the oil stops coming out. Second, clean up the oil that has already gotten out. Let's say it is 20,000 barrels a day (capturing 50% right), then that's equal to the Exxon Valdez every 13 days. The spill happened on April 20, 2010 so to date; we have 3 times the oil released from the Exxon Valdez.
The Big Picture
What's frustrating here is the lack of effort to really get everything under control on the clean up side. I found this video... careful, the narrator talks like an oil roustabout...
The point is there is a way to contain oil and there is a way it was done. They are different.
In this case, the solution looks like the video above should look, proper Booming would help. Several attempts to cap the well haven't worked. The cap they're putting on now is doing better. 6000 Barrels captured the first day, 10,000 captured the second day.
They are drilling a relief well but that won't be done until August so more oil leaking until then.
Containment just hasn't worked out. The shrimp are dead, the clams are dead, the fish are dead and the birds are dying.
I'm the last guy you'd hear calling for regulation. I believe in Laissez-faire (let it be) Capitalism, not what recently happened to our economy, that wasn't Capitalism, that was gluttony. I believe Capitalism means you don't destroy the market you make your income from.
Anyway, Canada says if you want to drill off shore, you need to start a relief well at the same time. This way, we're not waiting 4 months to get a spill contained. Also, I think BP shouldn't have been in charge of the clean up...
I have a daughter. If I say to my daughter, "go clean your room and get it as clean as you think it should be," it will remain a mess. She's not the one to have in charge of cleaning. In her case, I need to stand in the door way of her room and say "pick that up, put it in the dirty clothes, pick that up and now put it in the kitchen sink." It's just her way; she's not motivated to help me have a clean house.
I don't think BP Oil is motivated to make sure we have a clean gulf. I think it would have made more sense for our President Obama to put together a real team of experts who were motivated to do the job right and we might not have this to show for our efforts.
What's the point?
I'm not saying I did everything exactly right with my server crash. I could have been up 1-2 days quicker if I were already prepared. It wouldn't have happened if the alert system would have worked correctly but it happened and the only reason it got fixed in 5 days is because of quick action and willingness to spend to money to fix the problem sooner rather than later. Had I tried to cut corners, I'd have several clients still down and several clients with holes in their data since the last SQL Server backup.
In the case of the BP Oil spill, I think they are doing everything they can to stop the flow of oil... after all, every barrel of oil spilled is a barrel they can't sell. I think their efforts at containment were less than honorable and as much as I hate to criticize the President, I can't think of anyone else who had the power to take control, take responsibility and stop the oil from reaching the shores. He could have... he still has the authority to pull together his own team and get this handled correctly then send the bill to BP Oil.
In this situation, No one is going to win. I'm living on the Pacific Coast and my life will still be adversely affected. It punctuates the point that ecological waste affects everyone, Sorry!
Make the Conversation Green