Saying things like "ahead of the team passing out" in their latest status updates doesn't exactly give confidence that they have either good communications or engineering leadership.
If everyone has been up 24hrs then you will get status updates like that.
I guess it is all hands on deck and there isn't a large enough team to do rotations of shifts? Or the issue is very complex and you need all your best guys around to fix it.
Yikes.
As someone who runs a high availability website, I can feel for them, but it is really bad to be offline this long. I would look to make some changes to that team or address an unstaffing issue or something. This should not happen. This should have been preventable/foreseeable and a risk mitigation strategy could have been in place.
From what I suspect, we are talking about some backend issue that they are handling. Throughout my career I've never seen a "rotating shifts team" that can handle situations like this. It's usually one guy trying to fix it and a couple of others hanging in the office with a cup of coffee and giving advice on what should the company do next or being forced to update the status page.
It's scary. Why not just revert to the old engine when things started to fell out of place? Having the exchange down for that long is bound to have a big backlash when it comes back online.
They should have put a Kraken 2.0 trade engine alongside the first one, and moved people gradually there. It doesn't matter how confident they were with the upgrade before it happened, it's crypto, everything is new. A few lines of wrong code and you can lock millions of dollars in multi-sig wallets.
I have most of my funds there because their eur SEPA transfers worked very well. I really hope they can get back in shape after it comes back online.
Their downtime and non usable site was so bad already that downtime is almost as inconvenient as the working site was before.
Now at least they are fixing it. They held back the update for months due to testing. They have to take the jump and after x hours of downtime the damage is done, so fixing it once and for all instead of rolling back might actually be the better solution.
I notice the "upgrade is coming" notice on their site last week, but I don't have a Kraken account, I was just interested in learning more about crypto-currency trading. Now they made me curious about their platform, for a purely technical perspective.
As you say, why didn't they just revert? Are they not able to? What are the steps in their system upgrade? Are they moving to new hardware? What's their setup like? What software are they running (custom written surely, but what language, which database technologies?)
Incidents like this make me curious, and I would love to read the post mortem on something like this.
Explaining why an upgrade didn't work doesn't compromise security. I'd guess moving the data didn't work or they corrupted a database and don't know how to repair it.. Explaining that you don't have proper data backups in place can be embarrassing but with a post mortem you can at least get some trust back (like Gitlab's incident).
Not explaining why you're offline for 24h doesn't help people to trust you
If they corrupted a live database and are not able to recover it they are in a world of hurt. While it is bad form, many people keep their coins on the exchanges and even if the bulk of an individuals coins are offline, they still likely have at least a small amount on their for trading.
If a table that connects user accounts to kraken owned wallets is corrupted and not recoverable people will be out millions. For some that would be the equivalent of your 401k issuing a post mortem for losing all of your retirement.
If this worst case scenario happened they are likely in severe damage control.
Most likely explanation though is that things are just taking longer than expected to upgrade what is by all measures likely a very technical and convoluted system.
It boggles my mind that it's in the state it is. Some kid in his basement knows you do staging rollouts and build in parallel, this is seriously the most basic IT knowledge in existence - and yet they still screwed it up, now over 24 hours later.
Whoever their CTO is, is clearly the worst kind of incompetent and the team is the most amateur I have ever seen in my 20 years of Internet systems management.
Uber is great, but the biggest cost is that you need to pay for the services of someone driving for you. I live in Montreal and regularly use http://communauto.com/ They have two services, 1) for ~$2,50/hour + $0,10/km, where you need to make a reservation, you get the car from one specific parking lot and have to return it there. It's useful only if you plan in advance, if you haven't reserved a car for the weekend by Wednesday forget about it.
The 2) service though you just swipe your metro card on the panel for an auto-mobile car parked somewhere, drive for $0,30/minute and leave the car anywhere (in some very big zones in the cities). It's exactly like https://www.car2go.com/ except their auto-mobile vehicles (hybrids and electrics) have backseat, so we can go around with kids (whereas you can't with a Smart).
I commute by bicycle to work, so we use this only on the weekends. It takes around $7 to go from our place to the center, while public transport would be $5, so it's really great for the convenience and speed.
By far it really beats owning a car in Montreal in cost and convenience. Parking space is limited and very expensive ($8 for 2 hours in city center). If you have a car you have also to keep changing parking even if you don't use it because there are very few buildings with garage and they clean the streets at least once or twice a week for the entire year. I've been car-free for over 20 years, it's a huge relief to not have to worry about taxes, maintenance, robbery, etc etc that comes with the responsibility of owning a car.
Come self-driving cars, we might have only some very few people that will keep owning their own vehicles due to some nostalgia feeling.
I'm saving up to upgrade my PC (AMD HSA on Kaveri + DDR4) because of SteamOS (amongst other things) instead of getting a next-gen console. And I know a few other people doing that as well.
AMD needs to move to 14/16nm FinFET in 2015 as soon as possible (like Nvidia is doing). Everything is great on the GPU side, especially with all the added parallelism features, but they really need to close up that gap as much as possible with Intel on the CPU side. They can't do that by being 2 process nodes behind Intel, no matter how much they will innovate on the CPU side in the next few years, too.
If they try to be on the most cutting edge process node as early as possible (that's available to them), that could make them quite competitive and appealing, especially as all-in-one solution for gamers. I'd certainly consider them myself, too, but not until 2015, and until they jump to 14/16nm FinFET.
I am doing the same, once the OS gets released and I can buy controllers I will be hooking it up as the main system. The thing steam gets right is that everyone in the house who uses games has a great experience, from 10 - 35 years old it is extremely friendly and useful.
I have been using Steam on Linux and once they get a few more titles final (like CS: GO, they already have DOTA) it removes any need for Windows (home life isn't very Office centric).
I've also been doing this. I have a pretty beefy windows machine that I have configured to launch into big picture mode on boot with an Xbox 360 controller and the experience is about 85% there. It is kind of a pain in the ass having a controller, a keyboard, and a trackball in my living room in addition to the TV remote stuff.
It's also annoying to have a full desktop PC in the living room. When game streaming lands in Steam I'll be able to move that entire box into my office and then just have a simple front end Steam Machine handling driving the TV. People keep posting pics of the streaming UI but they haven't turned it on yet.
For my "front end" machines I am using 2 Ubuntu laptops with background auto updating turned on, steam in big picture mode on boot, and either an Xbox controller or a Logitech 710. I am thinking those will hold me over until Valve starts publishing the case CAD files and I'll be able to build my own proper machine.
Yeah, I agree. I was a 3rd country indie and strongly believer of UGC. I still am, but not in the mod sense. Modding is useful only if you want to break into the games industry, but that's it. And only the 0.1% of the good mods generate good game developers.
On the other hand, if you have "fun" tools for users to create content, they WILL create it. Like in Little Big Planet, ModNation Racers and others. It's amazingly hard to make a game, and amazingly harder to make a tool that is fun to use. And with this "fun tool" obviously you strip down all the publishing barriers (technical and legal) that "common game development" have.
I'm really beginning to think that the Snowden leaks came up too late, and the "intelligence-industrial-complex" might already be too big to dismantle.
They're funded through our 'representatives'. And (going by Sensenbrenner's statements lately) they'll have something to say about that when Congress rejoins.
Our representatives (no scare quotes needed) in the House came rather close to voting for a meat ax curtailment of the NSA the very first post-Snowden chance they got, and most importantly the vote didn't break on any of the usual lines like party or region.
I'd say it's way too soon to count out the normal political process, and there's recent history of the Congress doing the right thing: http://en.wikipedia.org/wiki/Church_Committee
But definitely not a good moment to make a joke.