Here at CCP, we’re always looking for ways in which we can increase our capacity to allow huge engagements such as the recent fight in UALX-3 to occur more smoothly and with no interruption.

We’ve always chosen the more challenging path when it comes to designing virtual worlds, focusing on single shard, open world sprawling environments where actions have consequences and where a single choice can ripple through the entire game environment, impacting the experiences of hundreds of thousands of players.

New Eden is a digital canvas, painted with fifteen years of incredible stories, including historic battles in 6VDT-H, M-OEE8, B-R5RB, Asakai, Nisuwa and many other systems that have become notorious for huge engagements and massive military action.

Every time an engagement of this magnitude occurs, we gather data that assists us in making changes that help us improve performance. This data also allows us to make informed choices regarding hardware upgrades such as those carried out when we replaced the entire infrastructure of the Tranquility cluster with the upgrade to Tranquility Tech III.

Despite upgrades however, since the start of 2018 we’ve experienced several issues that have affected Tranquility’s performance, not only in large scale engagements but in other areas too.

eH6iYo9

A timeline of issues

Initial problems began in November and December of 2017 with smaller database issues that initially caused minor extensions to daily downtime. This culminated in series of three substantial cluster crashes in late February and March in the run up to the March release. During this initial set of issues, we looked at various resolutions that ultimately did not resolve the core problem.

When we arrived at the deployment date for the March 2018 release, which included the release of the new chat backend, we once again suffered a series of database crashes on March 20th, directly on deployment day.

This highlighted an entirely separate issue with the chat system as pilots swarmed the server to log back in, overwhelming the new cloud-based chat cluster with connection requests in their enthusiasm to reconnect to New Eden. These events also highlighted several other issues that were rapidly resolved, as well as the scalability problem we encountered.

Stability continued to be an issue over the course of April, compounded by several more database issues that caused mass disconnects and reconnects of pilots, adding further load to the chat cluster.

To further complicate the situation, Tranquility was the target of a series of DDoS attacks between Fanfest 2018 and the release of EVE Online: Into The Abyss that added further stress to the system.

These attacks, alongside configuration issues with our DDoS mitigation service that didn’t take into account the chat cluster now being hosted outside Tranquility’s infrastructure meant that our mitigation caused connectivity issues with the chat service until the configuration could be rectified.

Thankfully, with the upgrade to SQL 2017 that we carried out in May, the database issues we were experiencing were resolved, and on that side of things there’s been much smoother sailing. The chat issues have still however persisted and are currently a primary focus for the team working on the new system.

Immediate performance issues were mostly resolved on the day of the deployment, and subsequent connection issues that were highlighted during DDoS attacks were resolved during that period too. While a significant number of issues have been taken care of, there are still lingering issues with channel inconsistency that are currently being worked on.

An additional level of complexity was added to the mix with the start of separate issues with the login servers. These problems intensified after the release of Into The Abyss and added further stress to the chat system.

Issues with the servers that host our Single Sign On system as well as the service that provides login tokens to the launcher resulted in times when pilots would see Tranquility online but couldn’t connect due to the launcher not receiving the correct information from our login servers.

All in all, a range issues across several different areas of our infrastructure managed by different specialist teams coincided to create the perfect storm, which we’re still currently weathering and resolving at the date of release of this blog.

And then of course, we add our bloodthirsty community into the mix!

So far, this summer has been one of the busiest we’ve seen in the last few years, and the conflict that’s ongoing has created more intense load than we were expecting on Tranquility.

With the addition of new Abyssal Deadspace systems, alongside several large fights and the ever-hungry needs of resource harvesting players who spend hour upon hour churning out the raw materials, components and blueprints to fuel the war machine, we’ve seen a far more active summer than we anticipated.

riaFODK

So what’s the plan?

We want to continue to provide the tools for you to wage conflicts of biblical proportions.

We also want to make sure that we continue to have room to grow, to accommodate larger engagements and more fierce fighting.

Regardless of the level of activity that’s occurred over the course of the summer, we’re aware that the impact of these issues on community sentiment has been severe.

We’re working several angles to make sure that the performance and reliability of Tranquility improves immediately, and quickly returns to the level our pilots expect and deserve.

Database Issues:

The upgrade to SQL 2017 has resolved the issues that we were experiencing with the EVE Online database, however we’re currently monitoring performance and looking at ways to further improve reliability and responsiveness.

DDoS Mitigation:

We’ve worked with our DDoS mitigation provider to ensure that when we next become the target of an attack, our infrastructure handles the traffic scrubbing process correctly and that our services aren’t adversely affected. Several configuration changes have been made in close collaboration with our partner to make sure that these issues do not re-occur.

Chat System Issues:

Work is ongoing to improve connectivity to the chat cluster and root out issues that are causing players to experience loss of connection. We’ve had two teams dedicated to looking into these issues over the course of May and June, and a third is now taking a fresh look into the more lingering connectivity and channel inconsistency issues.

Login Service Issues:

An engineering reliability task force has been formed at CCP to look at the issues with both the login service and the chat cluster. Their focus will be to look at improving reliability and reducing the occurrence of the issues that our pilots have been experiencing.

Adding More Hardware:

While more hardware isn’t always the right answer, adding more SOL nodes (the server blades that host solar systems in EVE) to Tranquility gives us more overhead to spread the load across more hardware.

This gives the cluster more breathing space in general and improves performance. While it doesn’t tackle the issue of load on a single node during a large-scale engagement, it allows us to assign other systems elsewhere and give these fights a little more horsepower and breathing room.

We usually have an additional Flex Chassis on standby, to swap out hardware that’s in service should we need to perform maintenance. Due to the high load on the cluster right now, we’ve placed this chassis and all its nodes into rotation to give more horsepower for the hosting of the Alliance Tournament systems as well as more spread for the load that higher activity than usual is causing.

Buying More Hardware:

In addition to adding existing servers that we’d usually use for maintenance swap outs, we’ve also spoken with our hardware partners, and we’re currently waiting on delivery of the following new hardware:

  • 4x SOL Nodes with 2x Intel Xeon Gold 5122 4C 105W 3.6GHz Processors.
  • 1x SOL Node with Intel Xeon Gold 6134 8C 130W 3.2GHz Processor for comparison.

These will be added to the cluster to test their performance and see how they handle dealing with so many spaceships, from there we'll be better informed on next steps.

kjpLffg

Where do we go from here?

After the extended downtime in March due to issues with the deployment of the new chat back end and the subsequent problems, we’ve issued a skillpoint gift to all those pilots who were active during the time of the March release along with the publishing of this blog.

Pilots will find that every character on accounts that were considered active on March 20th, 2018 (the date of the March release and the subsequent extended downtime), have been issued with a gift of 250,000 skillpoints.

(For reference, we define “active” as accounts that have logged in during the 30 days prior to a given date.)

While the war rages in New Eden for our pilots, the war for performance continues for us here at CCP.

We know that it’s a never-ending fight, but regardless, we’re pressed ever forward by the achievements and dedication of capsuleers across the globe.

We want the community to be aware that we fully understand concerns over reliability and performance and are working toward resolving these issues as soon as possible.

Our capsuleers deserve better, and we’re on a mission to make sure that performance and reliability return to the levels expected and continue to improve from there.

Our community is everything to us. We were ill prepared for such an increase in activity, and we let our pilots down.

For that, we sincerely apologise.

If there are any comments or questions regarding this blog, please feel free to jump on over to the comments thread on the official forums.

27 Comments

  1. thanks for informations

    August 2, 2018 at 12:36 Reply
  2. I’ve been exploring for a little bit for any high quality articles or weblog posts on this sort of house . Exploring in Yahoo I ultimately stumbled upon this site. Studying this information So i’m happy to show that I have a very just right uncanny feeling I discovered exactly what I needed. I such a lot for sure will make sure to don?t overlook this site and provides it a glance on a relentless basis.

    August 2, 2018 at 23:24 Reply
  3. Hope that these issues resolve quickly. I like each and every article posted by you guys. Keep up the good work.

    August 4, 2018 at 01:05 Reply
  4. thanks for all this info

    August 4, 2018 at 10:59 Reply
  5. Paragraph writing is also a fun, if you be acquainted with after that you can write otherwise it is difficult to write.

    August 4, 2018 at 12:37 Reply
  6. Hello are using WordPress for your site platform? I’m new to the blog world but I’m trying to get started and set up my own. Do you require any coding expertise to make your own blog? Any help would be really appreciated!

    August 4, 2018 at 13:01 Reply
  7. I’m impressed, I have to admit. Rarely do I come across a blog that’s equally educative and amusing, and without a doubt, you have hit the nail on the head. The issue is something that not enough folks are speaking intelligently about. Now i’m very happy that I found this during my hunt for something concerning this.

    August 5, 2018 at 11:37 Reply
  8. Hi! I know this is somewhat off topic but I was wondering which blog platform are you using for this site? I’m getting fed up of WordPress because I’ve had issues with hackers and I’m looking at alternatives for another platform. I would be great if you could point me in the direction of a good platform.

    August 5, 2018 at 23:59 Reply
  9. I simply want to mention I am just new to weblog and honestly loved this website. Very likely I’m likely to bookmark your blog . You absolutely come with really good posts. Kudos for sharing with us your website.

    August 6, 2018 at 03:10 Reply
  10. What’s up mates, its fantastic post concerning cultureand entirely explained, keep it up all the time.

    August 7, 2018 at 18:40 Reply
  11. Guau, increíble diseño de weblog! ¿Desde hace cuánto bloggeas? Usted hace que el blogging vistazo simple. ¡El vistazo total de su sitio es excelente, y mucho menos el contenido!

    August 9, 2018 at 08:16 Reply
  12. ¿Por qué razón los usuarios aún usan para leer periódicos cuando en este planeta tecnológico todo está libre en la red? ¡Volveré, puesto que he marcado esta página como preferida y lo he twitteado a mis seguidores!

    August 9, 2018 at 08:51 Reply
  13. I’ve been exploring for a little for any high quality articles or blog posts in this sort of space . Exploring in Yahoo I eventually stumbled upon this website. Reading this info So i’m satisfied to express that I have an incredibly excellent uncanny feeling I came upon exactly what I needed. I most for sure will make sure to do not forget this site and give it a look on a relentless basis.

    August 9, 2018 at 09:33 Reply
  14. I was curious if you ever considered changing the page layout of your site? Its very well written; I love what youve got to say. But maybe you could a little more in the way of content so people could connect with it better. Youve got an awful lot of text for only having 1 or two images. Maybe you could space it out better?

    August 10, 2018 at 04:29 Reply
  15. But you have to be prepared. It is not simple.

    August 12, 2018 at 10:52 Reply
  16. “Great awesome issues here. I am very happy to look your article. Thanks so much and i am having a look forward to contact you. Will you please drop me a mail?”

    August 15, 2018 at 13:47 Reply
  17. Hey There. I stumbled upon the web site the application of yahoo kumpulan youtube terbaik. That’s a actually beautifully written document. I’ll always book mark the item in addition to resume read more of one’s helpful facts. We appreciate you the actual submit. I’m going to unquestionably give back.

    August 16, 2018 at 01:10 Reply
  18. If your real friends know you as your nickname, use that nickname as your first name online. When you first friend someone, focus on making a personal comment that weaves connection.

    August 16, 2018 at 17:33 Reply
  19. Heya i am for the first time here. I found this board and I find It truly useful & it helped me out much. I hope to give something back and aid others like you helped me.

    August 17, 2018 at 06:07 Reply
  20. Thanks so much for the blog post.

    August 17, 2018 at 15:06 Reply
  21. Hello.This post was extremely interesting, especially since I was investigating for thoughts on this topic last Saturday.

    August 18, 2018 at 01:31 Reply
  22. Hey, thanks for the blog post.Much thanks again. Much obliged.

    August 18, 2018 at 05:30 Reply
  23. I do believe all the ideas you’ve introduced on your post. They’re really convincing and will certainly work. Still, the posts are very quick for newbies. May just you please prolong them a little from next time? Thank you for the post.

    August 18, 2018 at 05:57 Reply
  24. Im grateful for the blog.Much thanks again. Cool.

    August 18, 2018 at 14:24 Reply
  25. I’m still learning from you, while I’m making my way to the top as well. I absolutely liked reading all that is written on your blog.Keep the information coming. I liked it!

    August 18, 2018 at 15:44 Reply
  26. I carry on listening to the rumor speak about receiving boundless online grant applications so I have been looking around for the best site to get one. Could you advise me please, where could i acquire some?

    August 18, 2018 at 15:44 Reply
  27. MetroClick specializes in building completely interactive products like Photo Booth for rental or sale, Touch Screen Kiosks, Large Touch Screen Displays , Monitors, Digital Signages and experiences. With our own hardware production facility and in-house software development teams, we are able to achieve the highest level of customization and versatility for Photo Booths, Touch Screen Kiosks, Touch Screen Monitors and Digital Signage. Visit MetroClick at http://www.metroclick.com/ or , 121 Varick St, New York, NY 10013, +1 646-843-0888

    August 19, 2018 at 05:42 Reply

Leave a Reply