Website Crash

 January 8th,

Our database provider had an outage this evening.  The cascading effect is the website was not able process user requests and then crashed. I believe this lasted no more than 10 minutes.

I have taken steps move our database requirements in house.  This should prevent further disruptions due to factors out of my control. I will be transitioning tomorrow. There may be a momentary interruption in service.


May your games always be fun,
Pencils forever sharp,
and your rules books at the ready.

G Pa Dax (Brad)

Comments

Post a Comment

Popular posts from this blog

I surrender :(

Image
  I have worked feverishly over the last 6 weeks trying to make this new setup work.  No matter what I try I can't fix the poor performance some people are experiencing. I lay awake at night racking my brain for new things to try and trust me I have tried a great many things.  I have to recognize the new infrastructure I created just doesn't work well.  The improvements I promised the users just didn't pan out.  Actually things are worse than what I upgraded from. So what now?  I am going back to the previous cloud provider I was using.  It is an infrastructure I know works. I will try to use their environment more efficiently to try to gain the performance that I was wishing for. I have been giving a lot of thought on how to do this with zero downtime.  I think I can make this happen without nothing but a "Disconnection" message in your game window.  I will update the Discord announcements with all the details.  I am hoping to do this b...
Read more

Sunshine and Rainbows

Image
  Ok, well it looks like the migration worked.  It took just 90 min to move all the user accounts and update the user data.  There were a few bugs that I was able to hammer out quickly. It was a long road to get here but we are here. Early reports are encouraging.  We are seeing 3 to 4 times performance improvements.   Again I just want to thank you all for hanging in there with me on this journey and having faith that it would work eventually. May your games always be fun, Pencils forever sharp, and your rules books at the ready. G Pa Dax Brad
Read more

Smoking gun and a dead chicken.

Image
I have been working non stop on a lingering issue. During peak times some players/gms are not able to load their worlds. What has made this a challenge to debug is that it was so random.  People would show up in Discord and look for a solution. I tried my best to cycle through all the known issues, but in the end we couldn't solve it.  Games where cancelled and dreams of world domination by the players were dashed. A customer came into discord with the same issue. I was fortunate in that this person was not in the middle of a game and was able to stick around and help me do some deep diving.  So here is the funny thing about running a production environment.  In order to make sure things are  healthy and to help diagnose issues like this we set up monitoring tools.  These tools collect a great deal of statistics and forward them to a collection point. Then we can use fancy graphs to look at the data. Here is what one graph would look like. Pretty cool eh?...
Read more