Here at CCP, we’re always looking for ways in which we can increase our capacity to allow huge engagements such as the recent fight in UALX-3 to occur more smoothly and with no interruption.

We’ve always chosen the more challenging path when it comes to designing virtual worlds, focusing on single shard, open world sprawling environments where actions have consequences and where a single choice can ripple through the entire game environment, impacting the experiences of hundreds of thousands of players.

New Eden is a digital canvas, painted with fifteen years of incredible stories, including historic battles in 6VDT-H, M-OEE8, B-R5RB, Asakai, Nisuwa and many other systems that have become notorious for huge engagements and massive military action.

Every time an engagement of this magnitude occurs, we gather data that assists us in making changes that help us improve performance. This data also allows us to make informed choices regarding hardware upgrades such as those carried out when we replaced the entire infrastructure of the Tranquility cluster with the upgrade to Tranquility Tech III.

Despite upgrades however, since the start of 2018 we’ve experienced several issues that have affected Tranquility’s performance, not only in large scale engagements but in other areas too.

eH6iYo9

A timeline of issues

Initial problems began in November and December of 2017 with smaller database issues that initially caused minor extensions to daily downtime. This culminated in series of three substantial cluster crashes in late February and March in the run up to the March release. During this initial set of issues, we looked at various resolutions that ultimately did not resolve the core problem.

When we arrived at the deployment date for the March 2018 release, which included the release of the new chat backend, we once again suffered a series of database crashes on March 20th, directly on deployment day.

This highlighted an entirely separate issue with the chat system as pilots swarmed the server to log back in, overwhelming the new cloud-based chat cluster with connection requests in their enthusiasm to reconnect to New Eden. These events also highlighted several other issues that were rapidly resolved, as well as the scalability problem we encountered.

Stability continued to be an issue over the course of April, compounded by several more database issues that caused mass disconnects and reconnects of pilots, adding further load to the chat cluster.

To further complicate the situation, Tranquility was the target of a series of DDoS attacks between Fanfest 2018 and the release of EVE Online: Into The Abyss that added further stress to the system.

These attacks, alongside configuration issues with our DDoS mitigation service that didn’t take into account the chat cluster now being hosted outside Tranquility’s infrastructure meant that our mitigation caused connectivity issues with the chat service until the configuration could be rectified.

Thankfully, with the upgrade to SQL 2017 that we carried out in May, the database issues we were experiencing were resolved, and on that side of things there’s been much smoother sailing. The chat issues have still however persisted and are currently a primary focus for the team working on the new system.

Immediate performance issues were mostly resolved on the day of the deployment, and subsequent connection issues that were highlighted during DDoS attacks were resolved during that period too. While a significant number of issues have been taken care of, there are still lingering issues with channel inconsistency that are currently being worked on.

An additional level of complexity was added to the mix with the start of separate issues with the login servers. These problems intensified after the release of Into The Abyss and added further stress to the chat system.

Issues with the servers that host our Single Sign On system as well as the service that provides login tokens to the launcher resulted in times when pilots would see Tranquility online but couldn’t connect due to the launcher not receiving the correct information from our login servers.

All in all, a range issues across several different areas of our infrastructure managed by different specialist teams coincided to create the perfect storm, which we’re still currently weathering and resolving at the date of release of this blog.

And then of course, we add our bloodthirsty community into the mix!

So far, this summer has been one of the busiest we’ve seen in the last few years, and the conflict that’s ongoing has created more intense load than we were expecting on Tranquility.

With the addition of new Abyssal Deadspace systems, alongside several large fights and the ever-hungry needs of resource harvesting players who spend hour upon hour churning out the raw materials, components and blueprints to fuel the war machine, we’ve seen a far more active summer than we anticipated.

riaFODK

So what’s the plan?

We want to continue to provide the tools for you to wage conflicts of biblical proportions.

We also want to make sure that we continue to have room to grow, to accommodate larger engagements and more fierce fighting.

Regardless of the level of activity that’s occurred over the course of the summer, we’re aware that the impact of these issues on community sentiment has been severe.

We’re working several angles to make sure that the performance and reliability of Tranquility improves immediately, and quickly returns to the level our pilots expect and deserve.

Database Issues:

The upgrade to SQL 2017 has resolved the issues that we were experiencing with the EVE Online database, however we’re currently monitoring performance and looking at ways to further improve reliability and responsiveness.

DDoS Mitigation:

We’ve worked with our DDoS mitigation provider to ensure that when we next become the target of an attack, our infrastructure handles the traffic scrubbing process correctly and that our services aren’t adversely affected. Several configuration changes have been made in close collaboration with our partner to make sure that these issues do not re-occur.

Chat System Issues:

Work is ongoing to improve connectivity to the chat cluster and root out issues that are causing players to experience loss of connection. We’ve had two teams dedicated to looking into these issues over the course of May and June, and a third is now taking a fresh look into the more lingering connectivity and channel inconsistency issues.

Login Service Issues:

An engineering reliability task force has been formed at CCP to look at the issues with both the login service and the chat cluster. Their focus will be to look at improving reliability and reducing the occurrence of the issues that our pilots have been experiencing.

Adding More Hardware:

While more hardware isn’t always the right answer, adding more SOL nodes (the server blades that host solar systems in EVE) to Tranquility gives us more overhead to spread the load across more hardware.

This gives the cluster more breathing space in general and improves performance. While it doesn’t tackle the issue of load on a single node during a large-scale engagement, it allows us to assign other systems elsewhere and give these fights a little more horsepower and breathing room.

We usually have an additional Flex Chassis on standby, to swap out hardware that’s in service should we need to perform maintenance. Due to the high load on the cluster right now, we’ve placed this chassis and all its nodes into rotation to give more horsepower for the hosting of the Alliance Tournament systems as well as more spread for the load that higher activity than usual is causing.

Buying More Hardware:

In addition to adding existing servers that we’d usually use for maintenance swap outs, we’ve also spoken with our hardware partners, and we’re currently waiting on delivery of the following new hardware:

  • 4x SOL Nodes with 2x Intel Xeon Gold 5122 4C 105W 3.6GHz Processors.
  • 1x SOL Node with Intel Xeon Gold 6134 8C 130W 3.2GHz Processor for comparison.

These will be added to the cluster to test their performance and see how they handle dealing with so many spaceships, from there we'll be better informed on next steps.

kjpLffg

Where do we go from here?

After the extended downtime in March due to issues with the deployment of the new chat back end and the subsequent problems, we’ve issued a skillpoint gift to all those pilots who were active during the time of the March release along with the publishing of this blog.

Pilots will find that every character on accounts that were considered active on March 20th, 2018 (the date of the March release and the subsequent extended downtime), have been issued with a gift of 250,000 skillpoints.

(For reference, we define “active” as accounts that have logged in during the 30 days prior to a given date.)

While the war rages in New Eden for our pilots, the war for performance continues for us here at CCP.

We know that it’s a never-ending fight, but regardless, we’re pressed ever forward by the achievements and dedication of capsuleers across the globe.

We want the community to be aware that we fully understand concerns over reliability and performance and are working toward resolving these issues as soon as possible.

Our capsuleers deserve better, and we’re on a mission to make sure that performance and reliability return to the levels expected and continue to improve from there.

Our community is everything to us. We were ill prepared for such an increase in activity, and we let our pilots down.

For that, we sincerely apologise.

If there are any comments or questions regarding this blog, please feel free to jump on over to the comments thread on the official forums.

765 Comments

  1. qwioJV Well I sincerely enjoyed studying it. This post offered by you is very helpful for correct planning.

    December 16, 2018 at 23:22 Reply
  2. Hello there, I found your site via Google while searching for a related topic, your web site came up, it looks good. I’ve bookmarked it in my google bookmarks.

    December 17, 2018 at 02:11 Reply
  3. I intended to post you the very little word so as to say thanks a lot once again relating to the exceptional suggestions you have documented in this article. It’s shockingly open-handed with you to give unhampered all many people could possibly have marketed as an electronic book to earn some profit on their own, especially seeing that you could have tried it in the event you wanted. These good ideas also acted like the great way to comprehend many people have the same passion like mine to learn a little more when it comes to this matter. I’m sure there are lots of more pleasurable periods in the future for those who read carefully your blog.

    December 17, 2018 at 02:11 Reply
  4. Hello! I know this is kinda off topic but I was wondering if you knew where I could locate a captcha plugin for my comment form? I’m using the same blog platform as yours and I’m having difficulty finding one? Thanks a lot!

    December 17, 2018 at 02:26 Reply
  5. Hi there! Someone in my Myspace group shared this site with us so I came to check it out.
    I’m definitely enjoying the information. I’m book-marking and will be
    tweeting this to my followers! Fantastic blog and outstanding style and
    design.

    December 17, 2018 at 03:20 Reply
  6. I used to be suggested this website by way of my
    cousin. I am no longer sure whether or not this post is written by him as nobody else recognise such special about my difficulty.
    You’re amazing! Thank you!

    December 17, 2018 at 06:44 Reply
  7. I believe this is among the most significant info for me.
    And i am glad reading your article. However wanna observation on some basic things, The
    web site taste is perfect, the articles is really nice : D.
    Just right process, cheers

    December 17, 2018 at 06:53 Reply
  8. Türkiyenin en iyi seo uzmanı seodahisi, kurumsal seo hizmetinde en iyisi. Deneyin farkı görün. Google’da ilk sayfada lolmak için seodahisi.

    December 17, 2018 at 18:45 Reply
  9. Thanks so much for the article post. Great.

    December 17, 2018 at 22:26 Reply
  10. Wow, incredible blog format! How long have you been blogging for? you made blogging glance easy. The entire glance of your website is great, as neatly as the content material!

    December 17, 2018 at 23:12 Reply
  11. A round of applause for your blog. Great.

    December 18, 2018 at 08:39 Reply
  12. wow, awesome blog. Fantastic.

    December 18, 2018 at 14:42 Reply
  13. Thank you for your blog.Much thanks again. Will read on…

    December 18, 2018 at 20:03 Reply
  14. You cancould definitelycertainly see your enthusiasmexpertiseskills in thewithin the articlework you write. The arenaThe worldThe sector hopes for moreeven more passionate writers like yousuch as you who aren’tare not afraid to mentionto say how they believe. AlwaysAll the timeAt all times go afterfollow your heart.

    December 18, 2018 at 23:43 Reply
  15. Thanks for the blog post.Much thanks again. Really Great.

    December 19, 2018 at 00:00 Reply

Leave a Reply