Here at CCP, we’re always looking for ways in which we can increase our capacity to allow huge engagements such as the recent fight in UALX-3 to occur more smoothly and with no interruption.

We’ve always chosen the more challenging path when it comes to designing virtual worlds, focusing on single shard, open world sprawling environments where actions have consequences and where a single choice can ripple through the entire game environment, impacting the experiences of hundreds of thousands of players.

New Eden is a digital canvas, painted with fifteen years of incredible stories, including historic battles in 6VDT-H, M-OEE8, B-R5RB, Asakai, Nisuwa and many other systems that have become notorious for huge engagements and massive military action.

Every time an engagement of this magnitude occurs, we gather data that assists us in making changes that help us improve performance. This data also allows us to make informed choices regarding hardware upgrades such as those carried out when we replaced the entire infrastructure of the Tranquility cluster with the upgrade to Tranquility Tech III.

Despite upgrades however, since the start of 2018 we’ve experienced several issues that have affected Tranquility’s performance, not only in large scale engagements but in other areas too.


A timeline of issues

Initial problems began in November and December of 2017 with smaller database issues that initially caused minor extensions to daily downtime. This culminated in series of three substantial cluster crashes in late February and March in the run up to the March release. During this initial set of issues, we looked at various resolutions that ultimately did not resolve the core problem.

When we arrived at the deployment date for the March 2018 release, which included the release of the new chat backend, we once again suffered a series of database crashes on March 20th, directly on deployment day.

This highlighted an entirely separate issue with the chat system as pilots swarmed the server to log back in, overwhelming the new cloud-based chat cluster with connection requests in their enthusiasm to reconnect to New Eden. These events also highlighted several other issues that were rapidly resolved, as well as the scalability problem we encountered.

Stability continued to be an issue over the course of April, compounded by several more database issues that caused mass disconnects and reconnects of pilots, adding further load to the chat cluster.

To further complicate the situation, Tranquility was the target of a series of DDoS attacks between Fanfest 2018 and the release of EVE Online: Into The Abyss that added further stress to the system.

These attacks, alongside configuration issues with our DDoS mitigation service that didn’t take into account the chat cluster now being hosted outside Tranquility’s infrastructure meant that our mitigation caused connectivity issues with the chat service until the configuration could be rectified.

Thankfully, with the upgrade to SQL 2017 that we carried out in May, the database issues we were experiencing were resolved, and on that side of things there’s been much smoother sailing. The chat issues have still however persisted and are currently a primary focus for the team working on the new system.

Immediate performance issues were mostly resolved on the day of the deployment, and subsequent connection issues that were highlighted during DDoS attacks were resolved during that period too. While a significant number of issues have been taken care of, there are still lingering issues with channel inconsistency that are currently being worked on.

An additional level of complexity was added to the mix with the start of separate issues with the login servers. These problems intensified after the release of Into The Abyss and added further stress to the chat system.

Issues with the servers that host our Single Sign On system as well as the service that provides login tokens to the launcher resulted in times when pilots would see Tranquility online but couldn’t connect due to the launcher not receiving the correct information from our login servers.

All in all, a range issues across several different areas of our infrastructure managed by different specialist teams coincided to create the perfect storm, which we’re still currently weathering and resolving at the date of release of this blog.

And then of course, we add our bloodthirsty community into the mix!

So far, this summer has been one of the busiest we’ve seen in the last few years, and the conflict that’s ongoing has created more intense load than we were expecting on Tranquility.

With the addition of new Abyssal Deadspace systems, alongside several large fights and the ever-hungry needs of resource harvesting players who spend hour upon hour churning out the raw materials, components and blueprints to fuel the war machine, we’ve seen a far more active summer than we anticipated.


So what’s the plan?

We want to continue to provide the tools for you to wage conflicts of biblical proportions.

We also want to make sure that we continue to have room to grow, to accommodate larger engagements and more fierce fighting.

Regardless of the level of activity that’s occurred over the course of the summer, we’re aware that the impact of these issues on community sentiment has been severe.

We’re working several angles to make sure that the performance and reliability of Tranquility improves immediately, and quickly returns to the level our pilots expect and deserve.

Database Issues:

The upgrade to SQL 2017 has resolved the issues that we were experiencing with the EVE Online database, however we’re currently monitoring performance and looking at ways to further improve reliability and responsiveness.

DDoS Mitigation:

We’ve worked with our DDoS mitigation provider to ensure that when we next become the target of an attack, our infrastructure handles the traffic scrubbing process correctly and that our services aren’t adversely affected. Several configuration changes have been made in close collaboration with our partner to make sure that these issues do not re-occur.

Chat System Issues:

Work is ongoing to improve connectivity to the chat cluster and root out issues that are causing players to experience loss of connection. We’ve had two teams dedicated to looking into these issues over the course of May and June, and a third is now taking a fresh look into the more lingering connectivity and channel inconsistency issues.

Login Service Issues:

An engineering reliability task force has been formed at CCP to look at the issues with both the login service and the chat cluster. Their focus will be to look at improving reliability and reducing the occurrence of the issues that our pilots have been experiencing.

Adding More Hardware:

While more hardware isn’t always the right answer, adding more SOL nodes (the server blades that host solar systems in EVE) to Tranquility gives us more overhead to spread the load across more hardware.

This gives the cluster more breathing space in general and improves performance. While it doesn’t tackle the issue of load on a single node during a large-scale engagement, it allows us to assign other systems elsewhere and give these fights a little more horsepower and breathing room.

We usually have an additional Flex Chassis on standby, to swap out hardware that’s in service should we need to perform maintenance. Due to the high load on the cluster right now, we’ve placed this chassis and all its nodes into rotation to give more horsepower for the hosting of the Alliance Tournament systems as well as more spread for the load that higher activity than usual is causing.

Buying More Hardware:

In addition to adding existing servers that we’d usually use for maintenance swap outs, we’ve also spoken with our hardware partners, and we’re currently waiting on delivery of the following new hardware:

  • 4x SOL Nodes with 2x Intel Xeon Gold 5122 4C 105W 3.6GHz Processors.
  • 1x SOL Node with Intel Xeon Gold 6134 8C 130W 3.2GHz Processor for comparison.

These will be added to the cluster to test their performance and see how they handle dealing with so many spaceships, from there we'll be better informed on next steps.


Where do we go from here?

After the extended downtime in March due to issues with the deployment of the new chat back end and the subsequent problems, we’ve issued a skillpoint gift to all those pilots who were active during the time of the March release along with the publishing of this blog.

Pilots will find that every character on accounts that were considered active on March 20th, 2018 (the date of the March release and the subsequent extended downtime), have been issued with a gift of 250,000 skillpoints.

(For reference, we define “active” as accounts that have logged in during the 30 days prior to a given date.)

While the war rages in New Eden for our pilots, the war for performance continues for us here at CCP.

We know that it’s a never-ending fight, but regardless, we’re pressed ever forward by the achievements and dedication of capsuleers across the globe.

We want the community to be aware that we fully understand concerns over reliability and performance and are working toward resolving these issues as soon as possible.

Our capsuleers deserve better, and we’re on a mission to make sure that performance and reliability return to the levels expected and continue to improve from there.

Our community is everything to us. We were ill prepared for such an increase in activity, and we let our pilots down.

For that, we sincerely apologise.

If there are any comments or questions regarding this blog, please feel free to jump on over to the comments thread on the official forums.


  1. There as definately a lot to find out about this issue. I like all of the points you made.

    February 13, 2019 at 18:48 Reply
  2. Major thanks for the blog post.Really looking forward to read more. Awesome.

    February 14, 2019 at 01:16 Reply
  3. Needed to put you that very small word to give thanks over again for your striking tactics you’ve documented on this website. It’s really strangely open-handed of people like you to allow extensively exactly what many individuals would’ve made available as an electronic book to help make some money on their own, most importantly now that you might well have done it in case you wanted. Those pointers additionally worked like the good way to be sure that other people have similar dreams really like my personal own to see somewhat more with regards to this issue. I’m sure there are millions of more fun periods in the future for people who scan your blog post.

    February 14, 2019 at 03:02 Reply
  4. Fantastic web site. A lot of helpful info here. I¡¦m sending it to some buddies ans additionally sharing in delicious. And obviously, thanks for your sweat!

    February 14, 2019 at 03:03 Reply
  5. This is really interesting, You’re a very skilled blogger. I have joined your rss feed and look forward to seeking more of your fantastic post. Also, I’ve shared your website in my social networks!

    February 14, 2019 at 06:29 Reply
  6. Hiya, I’m really glad I have found this information. Today bloggers publish only about gossips and web and this is actually irritating. A good site with interesting content, this is what I need. Thank you for keeping this web site, I will be visiting it. Do you do newsletters? Cant find it.

    February 14, 2019 at 06:30 Reply
  7. Im grateful for the post.Really thank you! Want more.

    February 14, 2019 at 08:35 Reply
  8. Hello, i think that i saw you visited my website so i came to “return the favor”.I am attempting to find things to enhance my website!I suppose its ok to use a few of your ideas!!

    February 14, 2019 at 08:58 Reply
  9. I would like to express some thanks to you just for rescuing me from this difficulty. Just after looking out throughout the search engines and meeting strategies that were not helpful, I believed my entire life was over. Being alive without the presence of approaches to the problems you have sorted out as a result of the review is a critical case, as well as those that could have in a wrong way affected my career if I had not encountered your blog post. Your skills and kindness in handling every part was important. I don’t know what I would’ve done if I had not encountered such a thing like this. I can also now look forward to my future. Thanks a lot so much for the high quality and result oriented guide. I won’t be reluctant to suggest your web site to anyone who will need care on this problem.

    February 14, 2019 at 09:10 Reply
  10. I wanted to thank you for this very good read!! I definitely enjoyed every bit of it. I ave got you saved as a favorite to check out new things you post

    February 14, 2019 at 12:52 Reply
  11. This is one awesome blog article. Great.

    February 14, 2019 at 15:49 Reply
  12. your blog is really a walk-through for all of the information you wanted about this and didn at know who to ask. Glimpse here, and you all definitely discover it.

    February 14, 2019 at 20:22 Reply
  13. you have got an amazing weblog right here! would you wish to make some invite posts on my weblog?

    February 14, 2019 at 22:37 Reply
  14. I simply could not go away your website prior to suggesting that I extremely enjoyed the usual info a person provide for your visitors? Is gonna be again steadily to check out new posts.

    February 15, 2019 at 00:50 Reply
  15. Thank you in your curiosity in Funding Recommendation.

    February 15, 2019 at 01:09 Reply
  16. Fantastic blog.Thanks Again. Want more.

    February 15, 2019 at 11:41 Reply
  17. A monetary advisor is your planning associate.

    February 16, 2019 at 01:09 Reply
  18. I really liked your post. Fantastic.

    February 16, 2019 at 05:53 Reply
  19. All types of investments include certain risks.

    February 17, 2019 at 01:04 Reply
  20. Im grateful for the blog post.Much thanks again. Really Great.

    February 17, 2019 at 01:16 Reply
  21. Financial advisors enterprise is individuals enterprise.

    February 17, 2019 at 01:17 Reply
  22. Take into account investments that offer quick annuities.

    February 17, 2019 at 03:13 Reply
  23. I value the post. Awesome.

    February 18, 2019 at 12:53 Reply
  24. This page really has all of the info I needed concerning this subject and didn at know who to ask.

    February 18, 2019 at 13:37 Reply
  25. Now it’s time to pick your particular investments.

    February 19, 2019 at 02:45 Reply
  26. Thanks for sharing this very good write-up. Very interesting ideas! (as always, btw)

    February 19, 2019 at 10:41 Reply
  27. Really informative article. Want more.

    February 19, 2019 at 13:24 Reply
  28. Hey there, You’ve done an excellent job. I will definitely digg it and personally recommend to my
    friends. I’m sure they will be benefited from this website.

    February 19, 2019 at 14:58 Reply
  29. Thank you for the good writeup. It if truth be told used to be a enjoyment account it.
    Glance advanced to far introduced agreeable from you!

    By the way, how can we be in contact?

    February 19, 2019 at 17:24 Reply
  30. Thanks again for the blog post.Thanks Again. Much obliged.

    February 20, 2019 at 00:32 Reply
  31. Thanks for finally writing about >Dev Blog: Login, Database & Chat Problems – The War Continues!
    – Eve News24 <Loved it!

    February 20, 2019 at 02:32 Reply
  32. of the math that people should bear in mind, I want to add couple of other qualitative pieces

    February 20, 2019 at 07:27 Reply
  33. Thank you for your post.Really thank you! Awesome.

    February 20, 2019 at 13:27 Reply
  34. My relatives every time say that I am killing my time here at web, except I know I am getting know-how every
    day by reading thes pleasant posts.

    February 20, 2019 at 19:42 Reply
  35. My brother suggested I might like this blog. He was totally right. This post actually made my day. You can not imagine simply how much time I had spent for this info! Thanks!

    February 20, 2019 at 21:51 Reply
  36. I really like and appreciate your article.Really thank you! Will read on…

    February 21, 2019 at 03:12 Reply
  37. heyhello there and thank you for your informationinfo – I’veI have definitelycertainly picked up anythingsomething new from right here. I did however expertise somea fewseveral technical issuespoints using this web sitesitewebsite, sinceas I experienced to reload the siteweb sitewebsite manya lot oflots of times previous to I could get it to load properlycorrectly. I had been wondering if your hostingweb hostingweb host is OK? Not that I amI’m complaining, but sluggishslow loading instances times will very frequentlyoftensometimes affect your placement in google and cancould damage your high qualityqualityhigh-quality score if advertisingads and marketing with Adwords. AnywayWell I’mI am adding this RSS to my e-mailemail and cancould look out for a lotmuch more of your respective intriguingfascinatinginterestingexciting content. Make sureEnsure that you update this again soonvery soon.

    February 21, 2019 at 07:59 Reply
  38. Great article.Really thank you! Cool.

    February 21, 2019 at 09:46 Reply
  39. This is very interesting, You are a very skilled blogger.

    February 21, 2019 at 13:46 Reply
  40. This is one awesome article.Really looking forward to read more. Cool.

    February 21, 2019 at 15:13 Reply
  41. Great post. I used to be checking continuously this blog and I’m impressed!

    February 21, 2019 at 20:38 Reply
  42. I loved your blog article.Really thank you!

    February 21, 2019 at 23:31 Reply
  43. This awesome blog is really educating and also factual. I have discovered many handy tips out of this amazing blog. I ad love to visit it over and over again. Thanks a bunch!

    February 22, 2019 at 16:07 Reply
  44. Thanks for the article.Thanks Again.

    February 22, 2019 at 23:05 Reply
  45. Thanks for sharing your thoughts. I really appreciate your efforts and I will be waiting for your further
    post thank you once again.

    February 23, 2019 at 03:18 Reply
  46. Thanks again for the blog post.Much thanks again. Much obliged.

    February 23, 2019 at 11:10 Reply
  47. I delight in, lead to I found just what I was having a look for.

    You have ended my four day lengthy hunt! God Bless you man. Have a great
    day. Bye

    February 23, 2019 at 12:42 Reply
  48. wonderful points altogether, you simply won a logo new reader. What may you recommend about your submit that you simply made a few days ago? Any positive?

    February 23, 2019 at 14:25 Reply
  49. This is one awesome blog.Really thank you!

    February 23, 2019 at 15:22 Reply
  50. I don’t even know the way I finished up right here,
    but I assumed this submit was great. I don’t understand who you
    are however certainly you’re going to a famous blogger in the event you are not already.

    February 23, 2019 at 15:31 Reply

Leave a Reply