EN24 discord
sov map

Tranquility Woes

July 16, 2015

Tranquility (TQ), the live server EVE Online lives on, was down for an extended amount of time today. Long ago these extended downtimes were not unusual as expansions took several hours to install, but over the years the technology and the releases cycles made new expansions non-events. Nightly downtime itself was shrunk from one hour to 20 minutes. That is why today’s interruption is unusual. The last time EVE was offline this long was 2007, for the Tranquility expansion (barring DDOS attacks on the server that once took TQ down for 36 hours).

The nature of the problem was not publically released but look for a dev blog depicting the problem and the solution in the near future. CCP has always been forthright about technology issues, sharing information with the players since most of  them are associated with hi-tech sectors in the real world.

The Problem

Small post-expansion patches are routine, but when Tranquility would not start up properly several teams took notice. Initially, there was confusion over what would cause the nodes to fail repeatedly. After some efforts they opted to roll back to yesterday’s build, but the backup failed to start up as well. Both patches failed. Finally, CCP managed to use today’s patch with the server being stable.  After some testing they opened the doors to the public, and TQ accepted connections.

CJ-9hkOWgAQDOR4

CCP Nullarbor (Senior Programmer) surrounded by other Devs

Player speculation points to yesterday’s build being the first day of the new sovereignty system, and that today’s and yesterday’s builds could have both been faulty from the beginning. This would point coding for the the new sov mechanics as the problem. Another possibility could be hardware problems. CCP Falcon only said, “Sometimes Tranquility is just tricky.”

CCP wisely placed the server hardware in London, England instead of Iceland, where CCP is located. Island deliveries for broken equipment would be a great deal slower there.

CCP Falcon mentioned he would look into a convenience gift paid to players in the form of free skillpoints. This has been done in the past to sooth players, as well as on some anniversaries and holidays.

Whatever the reason, Tranquility is back and accepting connections, but careful not to jump in and start freighting your supplies all over to make up for lost time. CODE and others use these times to heavily hunt the trade routes. They’ve been waiting too.

 

 

Below is infotmation posted previously:

— CCP Falcon @ EVE-O Forums

Hey guys,

As you’re all aware, Tranquility is still offline after today’s scheduled daily downtime.

This is due to an issue with a code deployment today while patching for a few fixes in EVE Online. The issue caused the cluster to fail to start up correctly, and as such, Tranquility is currently unavailable for play for both EVE and DUST.

Our operations and development team are working on this issue as we speak, and we’re hoping to have Tranquility back online for you guys as soon as possible.

Right now, we don’t have an ETA for when the cluster will be back online and accepting connections, but we’ll keep you updated in this thread.

Cheers,

– F

UPDATES:

UPDATES:13:00 UTC – AFTER SEVERAL FAILED STARTUPS, WE ARE CURRENTLY INVESTIGATING A STARTUP ISSUE
13:30 UTC – INVESTIGATION IS STILL ONGOING – NO FURTHER ETA ON UPTIME FOR TRANQUILITY
13:37 UTC – STARTUP FAILED AGAIN – INVESTIGATION IS ONGOING
14:00 UTC – INVESTIGATION IS STILL ONGOING
14:30 UTC – WE ARE CURRENTLY IN THE PROCESS OF REBOOTING AFTER FAULTFINDING AND CHANGES
15:00 UTC – FURTHER ATTEMPTS TO START UP HAVE FAILED – INVESTIGATION IS ONGOING
15:30 UTC – INVESTIGATION STILL ONGOING – NO ETA ON SERVER UPTIME
16:00 UTC – A FURTHER FAILED STARTUP AFTER A NUMBER OF CHANGES – INVESTIGATION ONGOING
17:00 UTC – INVESTIGATIONS STILL ONGOING – NO ETA ON SERVER UPTIME
17:30 UTC – WE ARE CURRENTLY FAULTFINDING – NO ETA ON SERVER UPTIME
18:00 UTC – AFTER TESTING ON SINGULARITY WE WILL BE REBOOTING TQ TO TEST A FIX
18:30 UTC – WE ARE CURRENTLY STILL FAULTFINDING ON TQ – NO ETA FOR UPTIME AT THIS STAGE
19:00 UTC – WE ARE CURRENTLY TESTING A FIX FOR THE ISSUES ON TQ – NO ETA FOR UPTIME YET
19:30 UTC – TRANQUILITY IS STILL CURRENTLY BEING EXPERIMENTED ON – WE WILL CONTINUE TO UPDATE
20:00 UTC – PIZZA HAS BEEN ACQUIRED AND WE ARE CURRENTLY REFUELLING OUR DEVS!
20:30 UTC – DEVS REFUELLED – TESTING HAS RESUMED!
21:00 UTC – WE ARE CURRENTLY REBOOTING TQ AND WILL BE TESTING A FIX OVER THE NEXT 30 MIN
21:30 UTC – TESTING IS CURRENTLY ONGOING – NO ETA FOR UPTIME
22:00 UTC – WE ARE DEPLOYING A NEW BUILD TO TRANQUILITY FOR TESTING
22:40 UTC – TRANQUILITY IS NOW BACK ONLINE AND ACCEPTING CONNECTIONS!