Stability Update

 Hello,

I wanted to update the membership on where we are at with the transition to the new infrastructure.  To say it was bumpy is an understatement.   I think we are on the other side.  I have been running issue free from Friday morning to Saturday night, which are our busiest times.

I have been watching a lot of status graphs and error logs and everything is finally running at normal levels.  This is good news.  It was starting to get dis heartening and it didn't look like there was a solution.  Well thanks to google Foo I was able to source a different piece of software that was more suitable to the task.

The membership also found some bugs and I was able to squash them. So to recap some changes.

  • Backup file download. I had wanted to use a http download service to make it easier for people to download the backup zip file.  This proved to be problematic for very large files.  Some users had backup files over 20 gig in size!  To fix this, I decided to move the backup file into your home dir and make it available for you to download at any time using a sftp client like Filezilla.  This backup file does NOT count towards your storage quota.
  • When an account is deleted, the user data will be archived for 30 days. Just in case the account is deleted by accident.
  • The recent dnd5e update is causing game servers to crash mid way through the migration.  This is due to the migration requiring more ram that is available.  It doesn't happen to all users, but the ones that have large compendiums is where we are seeing the issue.  I don't have a way to accommodate this condition so I have been just helping people one at a time as they come in to discord.
  • The new software that I have installed, does have one draw back and that is it takes a few seconds longer to start up the game server.  It throws a weird error, just give it a second and hit reload. It should come right up. I am working to try to fix this condition
  • I have also put in more checks and balances on the backup and restore to ensure the success of both functions.  
  • There have been a few reports of high ping times / lag during game play.  I am still investigating.  To that end I have installed a few network monitors and hope to have a clearer picture of what is going on. I will keep you updated.
  • Due to a software error, the normal 3 hour idle time shutdown got set to 1 hour.  This has been fixed and apologies all around.
I am sure there were a few other changes but that is all I can think of for now.   Thanks for taking the time to have a read.  Stop by Discord and say hi. If you have any questions or thought I would love to hear it.

May your games always be fun,
Pencils forever sharp,
and your rules books at the ready.

G Pa Dax (Brad)

Comments

Post a Comment

Popular posts from this blog

I surrender :(

Image
  I have worked feverishly over the last 6 weeks trying to make this new setup work.  No matter what I try I can't fix the poor performance some people are experiencing. I lay awake at night racking my brain for new things to try and trust me I have tried a great many things.  I have to recognize the new infrastructure I created just doesn't work well.  The improvements I promised the users just didn't pan out.  Actually things are worse than what I upgraded from. So what now?  I am going back to the previous cloud provider I was using.  It is an infrastructure I know works. I will try to use their environment more efficiently to try to gain the performance that I was wishing for. I have been giving a lot of thought on how to do this with zero downtime.  I think I can make this happen without nothing but a "Disconnection" message in your game window.  I will update the Discord announcements with all the details.  I am hoping to do this before next weekend, but tha
Read more

Sunshine and Rainbows

Image
  Ok, well it looks like the migration worked.  It took just 90 min to move all the user accounts and update the user data.  There were a few bugs that I was able to hammer out quickly. It was a long road to get here but we are here. Early reports are encouraging.  We are seeing 3 to 4 times performance improvements.   Again I just want to thank you all for hanging in there with me on this journey and having faith that it would work eventually. May your games always be fun, Pencils forever sharp, and your rules books at the ready. G Pa Dax Brad
Read more

Smoking gun and a dead chicken.

Image
I have been working non stop on a lingering issue. During peak times some players/gms are not able to load their worlds. What has made this a challenge to debug is that it was so random.  People would show up in Discord and look for a solution. I tried my best to cycle through all the known issues, but in the end we couldn't solve it.  Games where cancelled and dreams of world domination by the players were dashed. A customer came into discord with the same issue. I was fortunate in that this person was not in the middle of a game and was able to stick around and help me do some deep diving.  So here is the funny thing about running a production environment.  In order to make sure things are  healthy and to help diagnose issues like this we set up monitoring tools.  These tools collect a great deal of statistics and forward them to a collection point. Then we can use fancy graphs to look at the data. Here is what one graph would look like. Pretty cool eh?   While I had the customer
Read more