About the recent site outage

35 posts / 0 new
Last post
About the recent site outage

So I said I would post a few more details about why the site went down on the 27th and 28th.

A little bit of background info... or: what I do from day to day

As a little bit of background, Pixel Scrapper, like every other website in the world, runs off of a nuts and bolts computer (or server), much like one you probably have at home, but a bit more powerful, and hooked up with a super-fast connection to the web so that it can support the thousands of visits it receives every day. It lives in Germany, but I manage it via an SSH connection from our home in Corvallis, Oregon. Our server is what's called an "unmanaged server", which means the people in Germany supply us with the physical computer, but everything else is up to us: installing the operating system (Linux) and all the programs necessary to serve a website, setting up all the security measures to prevent it being hacked, and monitoring it from day to day. Keeping a website like this up and running is a pretty big job on the systems administration side, and requires a decent bit of technical expertise with things like Linux, Apache, MySQL, and such. Generally you have a systems administrator to handle your server(s), and that's the person's full time job. We can't afford a systems administrator at the moment though, so I handle all of that along with fixing bugs and implementing new features on the website, and helping Marisa with support requests. As you can see, Pixel Scrapper keeps me and Marisa pretty busy, as we're both pretty much holding down what would normally be 2-3 separate full-time jobs each smiley . We like it though, because the work is self-directed and interesting, and we really enjoy empowering others to be creative; also, you guys are just a great community to be a part of.

What went wrong

So I guess that was a pretty big tangent, but when considering website outages I think it helps to understand a bit about what goes into keeping a site up and running. What happened on the 27th was that one of the two hard drives that hold all of our server data went belly up. We're prepared for that situation, which is why we have two hard drives: each drive is an exact replica of the other, so if one of them fails, it's not that big of a deal. It does take a little while to get the drive replaced, though, and copy over all the data from the good drive, so the site has to go offline for a while for that to happen. In this case, there were also a few complications involved, and it wasn't entirely clear whether the failed hard drive was causing all the problems: some cached data became corrupted, and the site started behaving strangely. To mitigate the risk of problems down the road, I wanted to make a thorough investigation, and also fall back on a database backup that was guaranteed not to have any corrupt data hidden in a back corner. For these reasons that site was offline a bit longer than expected, and because of the database rollback, a few pieces of content that had been posted to the site were lost (a couple of layouts, a couple of forum posts, and a few comments).

Minimizing fallout

As I've said in the past, Marisa and I care very much about all of the user content that is posted to this site. All of the content is backed up on multiple hard drives (we have more than the two in our server), and we take regular database snapshots to try and insure that we don't lose anything; rolling back the database is always a last resort when we face a problem. Unfortunately, in some cases, it's just not worth the risk of having corrupted content in the database somewhere, and we need to roll things back a few hours. The percentage of site content that is lost in these cases is very, very small, but we know that that fact is little comfort to the person who loses their long, thought-out forum post, or the twenty layouts they just uploaded.

We will continue to do everything we can to minimize the risk of content loss, and we hope that these database rollbacks will become even less frequent in the future. The fact remains, however, that nothing posted on this website, or any website (or saved on your computer for that matter) is 100% guaranteed not to be lost. We can guarantee it about 99.x%, but not 100%.

To minimize the risk of losing content that you work hard to create, I always recommend having multiple backups stored in multiple locations. If you are making a long and thoughtful forum post, compose your post in a word processor on your computer, and then copy and paste it once you are finished: this way you have a backup copy on your computer. Even better, compose your post in Google Docs, where it's automatically saved as you type. Before a forum post even gets saved to our website, there is a risk that your computer or browser might crash, in which case your work is lost before anyone even has a chance to see it (I'm sure many of you have experienced that). The same goes for long replies or comments. And when it comes to uploading layouts, always make sure that you have a copy of your layout on your computer (preferably on multiple hard drives): never post to Pixel Scrapper and then delete the layout from your computer. If you're uploading dozens of layouts, spread your uploads over a few days, rather than posting them all at once.

I hope these tips will help you preserve any and all of your work in the future, and I hope we can make incidents like these less and less frequent on Pixel Scrapper. Thank you always for your understanding and support as we try and keep up with all there is to do while running this website! When server issues like this come up, it puts me back several days with regards to my development plans for new features and such, and I hope you won't hold that against me.

As usual, we will be sending out extra download credits to everyone to make up for the people who missed their daily credits during the site outage: these will roll out to everyone over the next 24 hours. I've also modified our daily credit system, so that in the future it will automatically play "catch up" for anyone who missed their credits.

All the best.

-Jordan

Thank you!

wow first time I've been able to get on in 3 days... I'm ecstatic!!!
I am so so sorry it has been so much work behind the scenes, but very glad you have gotten us more or less back to normal.

just a side note (in case no one has mentioned it): I did notice on my screen... I'm at like 8pt or less on the size of the site, fonts and everything is very teeny tiny for me. I had to enlarge it with control (+) about 3 times but off course that made every other site I had open in google chrome humongous... LoL

Glad to hear things are back to normal... I missed getting on the forum and catching up on reading the posts, but so happy you are back up and running. Thanks for the extra DC... now to decide what I want to download. LOL

Thank you for all your hard work in keeping this site working smiley

I'm not a techie but my hubby is. Now that we've started our own IT biz I'm learning alot. 6 Months ago I would not have understood a word you just said but about 3 weeks ago one of our clients also had this problem with their server. The mirror drive went wonky and the drive had to be replaced and I got to help and learn so thanks for giving us the behing the scenes info because we all get to learn new things. smiley

The things I take for granted...I had no idea so much was involved in keeping the site up and running. Wow! Thanks for the overview and thanks as well for taking on the hugely appreciated task of providing so much for so many for so little. You all are wonderful smiley

Thank you for all you do. So greatly appreciated. smiley

Very interesting to see behind the scenes like that Jordan. Thanks for all the hard work!

Jordan,

I have to say something to you - well, this was said already, but I need to reinforce: YOU ARE A FANTASTIC WEBMASTER smiley Even being from a family with lots of IT guys (including my bro, that now works for our state govenment, managing a group that deals with data), I still learned a lot with your post! And I love it smiley

Well, an additional reason: Data backup is a very important thing for everyone these days, but it´s a still a very negleted issue. I neglet backing up my stuff a lot (yeah, sad smiley ) so, when someone write about backups in a place I read, I always remember it for some days smiley Which remembers me, in the next weeks I´ll have to handle with creating an external backup for our (me+my hubby) image databasis.

Sooooo, thank you for everything! For the devoted work on Pixel Scrapper, for telling a bit about how things work which helps us to learn something new, to remember that anything is safe (even paper documents aren´t - Once I lost a document for years, and just recently found it) and for the backup hints... you know what? I´ll start to use google docs for long posts, I think it´s a great idea smiley

I know I'm a geek...I enjoyed reading about the technical stuff! smiley You have my sympathies! And we appreciate all you do!

What's funny was, when it went down the 27th, I was just coming to post the issue in the support forum, as it had been sporadic a few times for me over the previous day!!!

Many thanks for all the hard work that goes on behind the scenes - much appreciated xxx

Thanks for all your hard work and effort!!

Wow...you do a LOT for us here at Pixel Scrapper, and we are THANKFUL!!! You and Marisa are a great team! Thanks for the backup info., I
forget to back things up all the time and I have two EHD's just in case, so there is no excuse for me. I think I'll get right on that this week!!
THANKS FOR EVERYTHING YOU DO JORDAN AND MARISA!!!!

Thank you for all your hard work! It's highly appreciated!!! I really missed being able to go to Pixel Scrapper, since it's part of my daily routine smiley. I understand how much work it is to keep a site like this up and running and these problems you've experienced remind me to also make back-ups soon! You guys are amazing!!!

Thanks so much for your hard work. I appreciate the explanation too. I may not get all the technical details but I can see that y'all really put a lot of time, work, and love into this site.

hi thank u.

thank u so much. smiley

THanks so much for all you two do!! My hubby is also in IT so I understand exactly what you said!!! Lol

Things happen Jordan and i knew you were working on things on your side,i am sure you needed a well deserved break from all the drama you had to endure,thank you for all the wise info,nothing is guarentee'd i sure understand that as i lost a huge external hard drive with everything on it at one time.
You are doing great and i appreciate all you and Marisa do for us around here,i am so happy to have found this site.

Love you MARISA & Jordan!!

Thank you very much Jordan and Marisa! Also for the explanation. I also know how much work and hours it takes to manage a website.
I almost feel guilty that I couldn't help (although I am not a IT girl) smiley This site is such a great one and I love the community idea and all the layouts that people scrap.

thank you Jordan! I'm sure that you both are doing your very best, and we SOOOo appreciate it! smiley I think this site is awesome, and absolutely appreciate how hard you both work to make it so, and that you are so responsive and transparent when something goes a little wonky smiley best of luck with your current projects, we can certainly be patient! smiley

thanks you two! smiley

Thank you guys so much, you really do rock!

You and Marissa are appreciated so much as we know you have to take care of so my things so I'll just borrow Taronna's Great words and say Thanks as well for taking on the Hugely Appreciated Task of providing so much for So Many for So Little.
Thanks also for keeping us informed, another way you show us how wonderful you are.

Well, thank you. You guys are extremely giving and it is wonderful to see. I don't know many people that are like you guys.

Thank you again!

You rock, Jordan!!!! Thank you for the thoughtful explanation. I really appreciate all the work you and Marisa do on our behalf!!!

I am glad everything is back to normal.
Thanks for the extra DC I received yesterday, but... today I logged in and my DC was ... 0 ...
Is there something wrong?

Thanks,
Els

Thank you. You don't HAVE to give away freebies. So I will not look a gift horse in the mouth. Thanks again.

Glad things are back up and running. Thanks for all the hard work that you both put in for us. On another note, my DC points are playing up. I get 7 new DC points daily, I used those 7 points yesterday which left me with 21 points, but today my 7 free DC points have not been added, I still have only 21 points instead of 28 DC points. Today is the second day that I have not received my 7 free DC points, I really miss using up my daily free DC. points but I don't want to use my other points as I am saving them up. Hope all is OK. Again thanks for all that you both do, it is much appreciated.

Linda

Jordan, thank you so much for all your hard work on this website. I love being a part of this community and appreciate everything that you and Maris do smiley

I also appreciate everything you and Marisa do however I have seen you post that you want to know when something is not working.
I received the DC you posted for the day the site was down. Those came in la night. Unfortunately I have not received any points for yesterday or today. This site has become an addiction for me hahaha and I miss figuring out how to spend my daily points. Again Thank you for all you do and as someone said you don't have to give away anything so I am not complaining just informing.

Pages