Post Mortem or Rather Recap

 That was a long two days I have to admit.  I worked on this migration for the last year and a bit.  I thought I had everything covered.  I was hoping to have it all done in about an hour.  As some of you noticed, it was more like 5 hours.  Here are some things I encountered and where they stand as far resolution.

1) There was an issue with an invalid SSL/TLS certificate.  This is the thing that puts the lock symbol in your browser.  I put the wrong certificate up and didn't realize it had expired. People reported it and I was able to fix right away. 

Resolution: Bought a new certificate and put it up.

2) People were able to start their game server, however everything was slow as molasses. Very frustrating to say the least.

Resolution:  The file server needed tunning.  It has lots of capacity, however, the network was choking off the connections.  Change some config values and viola!

3) Random in game "Server Disconnects" notices.

Resolution: This one turned out to be tricky.  Because we play for hours in our games, sometimes doing a great deal of talking, the firewall can think that the connection is stale and will close it.  I  have made changes for it to better recognize long standing connections and leave them alone.

4) Handful of users games wouldn't start.

Resolution:  They were over their storage allotment and the game server couldn't start. They had to delete some data or upgrade their packages.  There was one case were the chat log was so huge that it was crashing the game. Remember folks delete or archive your chat logs after each game.

5) Credit card processing was delayed.

Resolution:  Remember the SSL issue in point one. Well I didn't put the new certificate on that application.  The credit card company wouldn't send any updates to a expired certificate. Doh! This may take a few days for the credit card company to catch up with all the transactions.

6) Delay in getting Backup/Restore active.

Resolution:  I had to re download all the old backup files from the old file server. This took time. Then I had to convert them from a tar archive to zip.  This would allow for people to be able to open their backups at home after they downloaded them. 

That was the bulk of the major issue that I encountered. There were small bugs in the website code that I killed rather quickly.  I am sure there still are more.  I just want to say thank you for being patient during this migration. It was a great deal of data to move and modify. Let me just say I buy hard drives in the Terabyte range. I am confident that it will be an improvement, it just make take a few days to settle and do the fine tuning that is always required.

May your games always be fun,
Pencils forever sharp,
and your rules books at the ready.

G Pa Dax (Brad)



Comments

Post a Comment

Popular posts from this blog

I surrender :(

Image
  I have worked feverishly over the last 6 weeks trying to make this new setup work.  No matter what I try I can't fix the poor performance some people are experiencing. I lay awake at night racking my brain for new things to try and trust me I have tried a great many things.  I have to recognize the new infrastructure I created just doesn't work well.  The improvements I promised the users just didn't pan out.  Actually things are worse than what I upgraded from. So what now?  I am going back to the previous cloud provider I was using.  It is an infrastructure I know works. I will try to use their environment more efficiently to try to gain the performance that I was wishing for. I have been giving a lot of thought on how to do this with zero downtime.  I think I can make this happen without nothing but a "Disconnection" message in your game window.  I will update the Discord announcements with all the details.  I am hoping to do this before next weekend, but tha
Read more

Sunshine and Rainbows

Image
  Ok, well it looks like the migration worked.  It took just 90 min to move all the user accounts and update the user data.  There were a few bugs that I was able to hammer out quickly. It was a long road to get here but we are here. Early reports are encouraging.  We are seeing 3 to 4 times performance improvements.   Again I just want to thank you all for hanging in there with me on this journey and having faith that it would work eventually. May your games always be fun, Pencils forever sharp, and your rules books at the ready. G Pa Dax Brad
Read more

Smoking gun and a dead chicken.

Image
I have been working non stop on a lingering issue. During peak times some players/gms are not able to load their worlds. What has made this a challenge to debug is that it was so random.  People would show up in Discord and look for a solution. I tried my best to cycle through all the known issues, but in the end we couldn't solve it.  Games where cancelled and dreams of world domination by the players were dashed. A customer came into discord with the same issue. I was fortunate in that this person was not in the middle of a game and was able to stick around and help me do some deep diving.  So here is the funny thing about running a production environment.  In order to make sure things are  healthy and to help diagnose issues like this we set up monitoring tools.  These tools collect a great deal of statistics and forward them to a collection point. Then we can use fancy graphs to look at the data. Here is what one graph would look like. Pretty cool eh?   While I had the customer
Read more