Site Crash

[Haltech]

Site crashed during the start of a backup shortly after 00:01 hrs last night. Generally, our system will shut down posting as the backup kicks off but unfortunately, someone had replied to a thread at the very moment the backup started. This causes the table to be open and the backup hung up, causing the DB to disconnect. No posts were lost and we suffered 5 hrs downtime.

[gtarman77787]

who was the faggot that posted as soon as it booted up?

can we flame him?
was it you?
was it phillycreamcheese?

i got all scared this morning at work when the site didnt load because i thought my work blocked the site

good thing its back!

[Fagwagon]

247 goes down all the time lol this is the first time ive heard of it happening here just another reason why we are better and they fucking blow.

[karl-hungus]

yeah, who can we blame for the meltdown?

[Aetherfury]

Yeah let's kick their ass (unless it was me in which case they should be forgiven)

When I saw the site was down my mind ceased to function due to lack of stuff to read and I turned into Russell Crowe from A Beautiful Mind, but at the end of the movie when he's all fcking crazy.

[802MS3]

LVM snapshots are your friend, Haltech!

literally only needs to lock the tables for about 10 seconds.

[Haltech]

Well you should set it up Opt but im not sold on it. All i really care about is the DB being backed up. That is the most important part of the site. With Snapshots, i need a LiveCD to do any type of restore... That isnt a bad thing except the server is 2100 miles away.

[802MS3]

Originally Posted by Haltech Well you should set it up Opt but im not sold on it. All i really care about is the DB being backed up. That is the most important part of the site. With Snapshots, i need a LiveCD to do any type of restore... That isnt a bad thing except the server is 2100 miles away.

Not necessarily...

Not sure how you have everything set up, but the way we have things set up is that we have 3 different master databases, with numerous slaves (We also have a master-master setup on one of our new applications/databases, but thats slightly different). We always backup from the slaves, never the masters. Every database server has the database on its own partition (logical volume). We stop the slave on the box, lock the tables, and run lvcreate with the -s (snapshot) and create a new logical volume. Then we gzip everything on there (which is really just the DB install) and push it to our NAS. We also grab the master status so the slave knows where to pick up from the master. Here is the perl script we use:

#!/usr/bin/perl
use DBIx::DWIW;
my $db_user = 'dbuser';
my $db_pass = 'password';
my $db_host = 'localhost';
my $db_name = 'mysql';
my $conn =  DBIx::DWIW->Connect(DB=>$db_name, User=>$db_user, Pass=>$db_pass, Host=>$db_host);
my $lvmbinpath = '/usr/sbin';
my $masterstatus = '';
printf "%-12s %s \n", "removing", scalar localtime time;
`$lvmbinpath/lvremove -f /dev/VolGroup00/dbbackup.1`;
printf "%-12s %s \n", "stopping slave", scalar localtime time;
system ("mysql -uroot -pmysqldb44 -e 'slave stop'");
sleep 10;
printf "%-12s %s \n", "flushing", scalar localtime time;
$conn->Execute("FLUSH TABLES");
$conn->Execute("FLUSH TABLES WITH READ LOCK");
$conn->Execute("FLUSH LOGS");
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime(time);
printf "%-12s %s \n", "snapping", scalar localtime time;
#   2. From another shell, create snapshot.
`$lvmbinpath/lvcreate -L30G -s -n dbbackup.1 /dev/VolGroup00/db`;
system ("mysql -uroot -pmysqldb44 -e 'show slave status\\G' >> master.status.latest");
system ("mysql -uroot -pmysqldb44 -e 'show master status\\G' >> master.status.latest");
printf "%-12s %s \n", "Master Status", scalar localtime time;
system ("tail -n 34 master.status.latest");
printf "%-12s %s \n", "unlocking", scalar localtime time;
#   3. From the first client, execute UNLOCK TABLES.
$conn->Execute("UNLOCK TABLES");
system ("mysql -u[user] -p[password] -e 'slave start'");
#   4. Show current snaphots
printf "%-12s %s \n", "LVM Status", scalar localtime time;
system ("$lvmbinpath/lvscan");
printf "%-12s %s \n", "done", scalar localtime time;
$conn->Disconnect;

We also have a script that rotates the backups - keeps the last weeks worth, one from last week, one from last month, etc.

Restoring is very easy, just untar the backup right onto the /db partition.

<3 Linux =P

Originally Posted by pat failman yeah, who can we blame for the meltdown?

it was Obama!

[felt]

It was me you motha fookers!!!

No just kidding!

[predapio]

Was that even English?^^^^^

[MS3-a(sore)ASS-rex]

Whoa opt that was some crazy shit! I have no clue what any ofthat meant....

But, BOT I almost had a goddamn heart attack when I couldn't check the PM I received in regards to my parts fml, but it's ok now...I missed you MSF hahaha

[Shanems3]

we all know it was philly so stop guessing