# Site Outage on Friday the 20th



## David Bott

Hello All...

I am SO SORRY about what happened today. Still not 100% sure what happened but the long and short of it is this.... We need to move to a slightly newer, but really actually still old, version of vBulletin...3.8.7. That version is at least still updated with patches and also supports TapaTalk better for mobile use of the site.

So, I hired someone to help do this. In looking to test an upgrade to ver 3.8.7, a copy was needed of the database. That is where something went wrong and it somehow crashed the "post" database table during the process. In trying to repair it, the /tmp area MySQL needed to use was too small for the 6GB "post" table. It was not known for some time that this was the cause of the failure to repair the database table. What took so long is that each time we tried we would not know for some time if it would fail or not as it takes a very long time to repair a table of this size. So each time we tried, each time we might have to wait an hour.

Fast forward....I then searched the Internet and hired an MySQL admin who was able to fix the file and here we are.

Before you ask, yes, we had a backup from 3AM. But I know some of you post a lot early in the day and I did not want to loose any of the posts. (I hope we did not.) So I went the repair route vs restoring which would have got the site back on-line sooner. 

That is more than you likely needed to know...but there it is more or less. 

All I can say is sorry. I am trying to do right by the site. When we get to the point, I will alert you when the site will be down to move to the new"er" vision.

Thank you all.

(BTW...It will be slower until it rebuilds the cache.)


----------



## DancnDude

Thank you! Having the communication about what's going on helps a lot and knowing you're interested in getting us up and running on more current software is a really nice thought that some effort is being placed on the site.


----------



## Turtleboy

Thanks, David!


----------



## whoknows55

I'm happy with anything that moves us to a fully patched and non-vulnerable version of vBulletin


----------



## bareyb

Appreciate the follow up, and all your hard work. Thanks.


----------



## David Bott

Oh, we are going only to ver 3.8 as it was really what most consider the most stable version. From ver 4 on it has been a nightmare from what I have read. (Though I have not done this in some time.)


----------



## waynomo

Thanks. I can only imagine how frustrating this was for you while on the road, knowing how many wanted to get on TCF, and have to wait an hour between attempts.


----------



## Arcady

Thanks for keeping us updated. Good luck with the updates!


----------



## murgatroyd

Thank you David. I appreciate all your hard work.


----------



## Adam1115

Thanks so much!!!


----------



## SullyND

Thanks for the update and for all you do. 

I'm seeing new ads on mobile, are those supposed to be there? (They take over the bottom 1/2" of the screen)


----------



## Mike Lang

Mobile skin or app or?
Got a screenshot?


----------



## murgatroyd

On the desktop, I'm now seeing a banner ad in between the last post in a thread and the quick reply box.


----------



## SullyND

Mike Lang said:


> Mobile skin or app or?
> Got a screenshot?


Mobile skin on iPhone. Let's see if this works.


----------



## David Bott

And this has what to do with the site outage thread?

Anyway, that skin looks like the main skin being used on a mobile device. So yes, ads would surely show. (I would not call that taking over 1/2 the screen. )

And yes, that ad has been there between the bottom of the thread and the Quick Reply box for some time. (1/1/15 actually)


----------



## SullyND

David Bott said:


> And this has what to do with the site outage thread?


I wasn't sure if it was something that got broke while trying to fix the site, which is why I asked. There was a point where some less, um, capable, had an issue with these types of ads appearing here unintentionally if I remember correctly. I can start a new thread for it if you like.



David Bott said:


> Anyway, that skin looks like the main skin being used on a mobile device. So yes, ads would surely show. (I would not call that taking over 1/2 the screen. )


Yeah, I guess it is the main skin, I thought it was mobile. I didn't say 1/2 the screen, I said 1/2 inch...



David Bott said:


> And yes, that ad has been there between the bottom of the thread and the Quick Reply box for some time. (1/1/15 actually)


I'm not talking about that ad - that's the purple one in my screen shot, right? (You can't beat). I'm talking about the black one at the bottom of the screen - " Get your tax refund faster with RushCard". It's intermittent, so I suspect it is coming from one of the main ads, perhaps the blank one at the top. It may be closer to 3/8" not that I look at it.

Thanks again for fixing the site.


----------



## unitron

You need to edit the thread title to indicate it was about the Friday outage and not the one this (Sunday) morning.

(although technically it wasn't a site outage, but just an individual forums outage)


----------



## David Bott

SullyND...Sorry, that last one was for the other posted under you. And yes, I could see maybe how you thought the site issue maybe was ad related, it was not. Yup, read that as 1/2 the screen.


----------



## StevesWeb

David, you're doing a very good job. The reliability is quite good here generally, and we all know problems do crop up.

The bottom line is this is a top tier forum, moderated very professionally, and we appreciate it being available.

Thanks.


----------



## SullyND

David Bott said:


> SullyND...Sorry, that last one was for the other posted under you.


Ah! No worries, thanks for all the work you are putting in to fixing up the site!


----------



## David Bott

Thanks guys. I had no idea what I was getting myself back into it seems.


----------



## scooterboy

SullyND said:


> I'm not talking about that ad - that's the purple one in my screen shot, right? (You can't beat). I'm talking about the black one at the bottom of the screen - " Get your tax refund faster with RushCard". It's intermittent, so I suspect it is coming from one of the main ads, perhaps the blank one at the top. It may be closer to 3/8" not that I look at it.


I know which type ads you're referring to. I also noticed the difference when the site came back up. It's the difference between this (the way it always was):










and this (takes up much more of the screen):










In my case, I'm on a 10" tablet and that larger ad took up the bottom 1/4 of my screen. Most of the time it's still the older smaller ad size, but a few times it comes up as the much larger version.

Timing could be coincidental, but as I said I never saw this until the site came back up this weekend.


----------



## David Bott

A new setup of Google DFP ad server went on-line last Thursdays evening with a new setup and new networks. So more than likely it now detects mobile devices and serves a mobile ad. Maybe they just started to serve. 

I can look to see what I can find, however, then mobile devices may not see ads as sometimes the standard ads would not be served to a mobile device.

I was referring to the ad above the Quick Reply Box when I was talking about how long ago THAT ONE was added.

When the site is migrated to a newer version TapaTalk becomes an option.


----------



## scooterboy

Cool - thanks for investigating, David. I know you're juggling a few balls at the moment, so feel free to put it on the back burner.


----------



## Hank

What is the problem with Tapatalk? Because I use it all the time with TCF and it appears to "just work". I'm using Tapatalk version 2.5.2 on iOS, and I really can't find anything that doesn't work on TCF. I'm reading and posting from it right now.


----------



## pteronaut

Hank said:


> What is the problem with Tapatalk? Because I use it all the time with TCF and it appears to "just work". I'm using Tapatalk version 2.5.2 on iOS, and I really can't find anything that doesn't work on TCF. I'm reading and posting from it right now.


Quote attribution doesn't work on Tapatalk with this board's vBulletin software version.

If it did, you'd see your username in the quote in this post in Tapatalk.

Hopefully, a newer version of vBulletin, will allow for better handling of posts by ignored users within Tapatalk also.


----------



## unitron

David Bott said:


> Thanks guys. I had no idea what I was getting myself back into it seems.


Thanks for fighting the good fight.


----------



## Hank

pteronaut said:


> Quote attribution doesn't work on Tapatalk with this board's vBulletin software version.
> 
> If it did, you'd see your username in the quote in this post in Tapatalk.


Eh, I see. I never thought that to be such an important feature. It's usually pretty easy to figure it out. <shrug>


----------



## waynomo

Hank said:


> Eh, I see. I never thought that to be such an important feature. It's usually pretty easy to figure it out. <shrug>


There are other short comings also. Yes, they tend to be minor.


----------



## David Bott

Well, still fighting the ongoing site outage that seems to happen at 3:00 each night when it backs up now. Thats new. It seems to be a left over from the main crash on Friday morning. Site has been down each morning starting around 3AM from Friday.

New server is ready but now a lot of testing will need to be done to see if about 3.8.8 running on a new box and also testing conversions etc. Then you have plugin that we may use that may or may not be available for that version. Fun times! (Right Hank? You know how it is. )


----------



## Rob Helmerichs

I take it this is going to be a full-season thread?


----------



## David Bott

Oh gosh....I hope not as I surely did not sign up for this.


----------



## dianebrat

David Bott said:


> Oh gosh....I hope not as I surely did not sign up for this.


You've got it backwards, usually it's "you break it, you buy it:


----------



## murgatroyd

David, since I'm going through this on a much smaller scale, trying to get my computer tuned up and working again, I really appreciate the effort you've been putting in.


----------



## Fofer

Thank you very much David! I really appreciate the time, energy and thoughtfulness you're putting into this.


----------



## David Bott

Thanks guys...we will get there. Please remember that when the time comes to make a move to a newer version, we will have put put up some somethings and maybe loose some things. Sometimes you need to just find where things are or something. You know how it goes.


----------



## David Bott

Hi All...

Well, it has been a "fun" last few days for the site in regards to the database and also login attempts by Bots. (no relation.)

On Friday morning when the site went down, for still unknown reasons, the messed up the POST table of the database. This is the largest of the tables that make up the site. As such, it takes time to fix it each time something happens.

Each morning from that crash, the site would again go down when it would backup. So at 3AM CT each morning, my phone would go off and I would say....What the F$#@!!!

Well FINELY it is FIXED!!!

You see I got a PM here the other day from member HANK. Hank has been a member of this site for a VERY long time and most of you know him. Turns out, Hank is a MySQL Jedi and on Sunday morning, Monday morning, and again Tuesday morning he would work with me on the database as we looked over "stuff" and he ran repairs.

The final fix was when it was thought that even though we were fixing the indexes each time, that the top header of the file was not being remade, but just reused, and thus was corrupted. So Hank made a new POST table, so it was fresh, and read all the data back in and indexed it.

3:00AM this morning came and I was not woken!!! (And either was Hank... by me.)

So, *THANK'S HANK* for the work on the site to get it figured out. (My wife thanks you also.)


----------



## Hank

You're welcome Dave. 

As I told David several times, it's the least I could do to give back to this crazy community after all these years.


----------



## Fofer

Thank you David. And thank you Hank. Your work is very, very much appreciated. :up:


----------



## Adam1115

Thanks, Hank! Nice!


----------



## sushikitten

:up:


----------



## whoknows55

Someone remind me to buy Hank a beer next time I see him.


----------



## Alfer

:up:


----------



## murgatroyd

Well done, Hank -- thanks!


----------



## DevdogAZ

Great job Dave and Hank!!


----------



## bareyb

Right on Hank. Thanks for sharing your Jedi Skilz. Yu da man.


----------



## StevesWeb

Yeah Hank!


----------



## SueAnn

Thanks Hank !


----------



## justen_m

I am going to kiss you. Is that ok?


----------



## waynomo

Thank you Hank and Dave! I know how frustrating figuring out these things can be.


----------



## Hank

justen_m said:


> I am going to kiss you. Is that ok?


Sure. That's got to better than waking up at 4am three days in a row!


----------



## rhuntington3

Nice work Hank and David! :up:


----------

