Human body is a complex system, with so many twists and turns. The more dramatic diet you are trying to introduce, the more careful you have to be.
There is no quick and easy solution, do the fasting thing and you'll lose weight in 10 days and then you can eat whatever you want. No. It's about the choices you make every day, it's about lifestyle. Eat healthy (most of the time), sleep well, rest, exercise, manage stress.. and you should be fine.
Yes, to some degree. But what people do when their dogs get too fat? They reduce the amount of food they give them, and after a while their weight goes down. There are many factors like stress, microbiome and sleep, but in the end it still boils down to calories in calories out.
People like to point to the edge cases, even though it is not helpful for 99% of the population. Let's be real: Almost no one ends up fat, if they are not eating badly (i.e. too much).
Actually for me fasting was quick and easy. I lost most of the weight within weeks and months. Fasting is much more powerful than your comment would indicate.
Theres a rate of fasting thats harmful though. On the TV show Alone, people are out in the woods for a few weeks to months and they end up just sloughing off weight. One contestant lost 86 pounds in 67 days, you'd practically see it fall off you each day. They do have to get medical checkups because losing all this weight too fast can put a lot of strain on your organs. A few people have been forced to leave the show because of starvation.
As someone who suffered from insomnia for most of his adult life, I honestly hate when people include "sleep well" in lifestyle choices. As it's something you can easily, or at all, control. On the plus side, diet specifically low carb diet finally cured it for me.
I get frustrated when people demand a special carve out for their situation in general advice. I mean, I guess they should have written:
Eat healthy (most of the time, unless you are in experiencing food scarcity, sorry), sleep well (unless you have insomnia, then try your best), rest (unless you have small children, then good luck), exercise (unless you have long COVID, there seems to be a trigger effect from exercise), manage stress (unless you are in a war-torn region, then I'm sending my prayers) .. and you should be fine (or your best you if fine isn't possible).
Sure I'm probably biased but I believe sleep quality is a little more difficult to control than the other points you listed. I can't even fathom consciously having bad sleep habits and not fixing them given how disrupting insomnia has been for me.
Honestly, my wife has insomnia and it is not only difficult to manage but it interacts with all the other issues: increases stress, makes eating healthy difficult (reduction of will power) and exercise is harder to recover from.
I'm sure it's frustrating to hear over and over again. But generalized advice doesn't and can't apply to everyone. Many people have extremely poor habits around sleep, and don't appreciate or dismiss the importance of it.
Maybe there's some underlying pathology in more cases than you will ever see acknowledged in traditional medical research (e.g. person is mildly addicted to being on a computer screen, or person has breathing problems that make it hard to sleep soundly). But that still doesn't make it bad advice on average and in general!
Looking from addiction perspective, TikTok is even worse. But, as the author said, it's more shallow, so it should be easier to move away from.
I hope that people (or at least majority) will find a way to move from this mindless fun and finally start using all the technology we have at our disposal in some more meaningful way.
Many recent boycotts and other activist causes have achieved critical mass through organizing on TikTok. A ton of content there is historical and educational. People misunderstand short-form as shallow and that has never sit right with me. I find creators on TikTok to often go deeper and communicate more clearly than the cacophony of text posts on reddit or Facebook.
Yeah I don't have numbers to back it up so I can't say for certain. I wouldn't agree it's negligible though, because I often am served educational content that's not an interest of mine (eg nutrition, DIY repair, writing). So the algorithm knows that content is desirable to some users, but is testing if I'm personally interested. Anecdotally I am constantly hearing friends talk about stuff they 'learned on TikTok'. Qualitatively, the content consistently seems more serious and useful than content I see on YouTube, Twitter, or Reddit. So my experience of TikTok has been that it's a mixture of entertainment and serious discussion. I think that balance is a large part of why it's so engaging.
Ok, I must take a step back. I have only my personal feeling to back the claim. And that doesn't mean much, considering the fact that I barely ever used TikTok.
So you might be right. My kids (6 and 8) told me a couple of times about life hacks that they saw on YouTube. And that's the only place they visit online, even that being restricted to less than an hour a day on average.
So yeah, new generations might be using that content a bit differently from what I thought.
I have a sensor LED lights in bathroom and in the kitchen. When I go to someone else's apartment now, usually it takes me a couple of seconds to realize why in the bathroom and it's dark. Haha
I don't know how people live without this. It's so convenient - you don't have to worry about turning on the light when you enter, turning it off when you leave. You're just in - out. At the same time, it saves money, because it's on only while you're in there.
Robot vacuum cleaner. Brought revolution to our lives. Since we got it, we forgot where our regular vacuum cleaner is.
Especially since dust bothers me - I set it up to clean while I'm not in (for example, every day at 10am, when we're at work), and voila! Or when you're not at home for a couple of days, you just trigger it an hour before getting home, and you get a clean apartment when you're home.
So, you get more time because you don't have to vacuum yourself and it does it while you're not in, so the whole process is seamless.
I love my kid, but it has flipped my life upside down. My boss is understanding. I WFH, we have a nanny for 4 hours a day, but still I start work at 1:30PM. I have almost no will to learn new things. I was working on a side project and have little will to continue. Our personal life is suffering as we are both tired.
We are considering a full time nanny which is going to hit our finances.
I have to say I've no idea what I'd do if I couldn't send our child to daycare. I work from home, but past experience has taught me that I cannot work if he's home.
On the rare times he's been sick I stay home and take care of him, and don't even pretend to work. Too much "Daddy play with me", "Daddy look at this", etc. Even if he's basically sofa/bed-bound and sleeping all day I can't work.
At least in Finland we're allowed to have 9ish days of paid leave a year to take care of a sick child.
Oh, man, I opened Pandora's box with this question. :)
I'm lucky to have boys of similar age, so they play each other whole day. On the other hand, they do need to eat, use toilet, peacekeeping from time to time, etc., so, it's a whole day duty. Luckily for me, I work for myself (a startup), so I can work anytime I want. For wife it's a bit different, she's 9-5 in normal circumstances, but with kids, it extends to the whole day.
It's not easy, but I love them.
Btw, my kids are 5 and 7 - to those of you with <4 years old kids, don't lose hope. As they grow and get more independent it gets easier. You'll have more time to get back to your side projects.
Look at this situation from the bright side - you are there for them now, when they need you the most, when their personalities are formed. Enjoy every moment, it will pass quicker than you think. Before you know it they'll be kicking you out of their room. And you'll be sad about it. ;)
I’m guessing your kid is quite young. Just to echo @ruffrey, hang in there, it gets better. They get easier and more independent, and you learn to adapt. Covid lockdown is a hell of a time to have a small child in the equation!
Totally feel you re side projects and learning new stuff though. There’s just no time and very little will. You need to find new ways to be effective with your time and energy, which can include making tough decisions about where you prioritise both.
I can only explain my system but it works like this: try to drink alcohol every day. I'm not sure if it reduces the covid risk, but it just feels right.
ugh childcare has been so tough. We went from 1 to 2 kids in August and cannot find a nanny or babysitter that doesn't have a covid exposure every other week.
Things that have worked for us...
With the kids... family walks and car rides.
After bedtimes... grown up dinner + korean dramas. wine.
We also have a couple games that are fun when we can sneak them in - 5 crowns (5 suit rummy card game), star crossed (2 player role-playing game that uses a jenga set), lots of trivia couples betting (no fact checking to see who is right about something without making a honey-do bet about it)
Imagine if all the companies in the world switched to this friendly, non-competitive mode? I heard something like that before... hmm, communism? You don't have to work (hard), you'll still get your paycheck.
Problem is - it doesn't work in the long term. It can work for a while, until your money runs out or someone more competitive comes along and blows you away.
I have a German layout keyboard atm (don't ask), and I barely even notice it.
~20 years ago I invested a lot of nerves into learning touch typing, so now I'm flawless with it.
But, let's talk about problems with touch typing - wrists for example. If you don't know touch typing, you move your hands more often, to do stuff. E.g. copy/paste, with touch typing, it's pinky + index finger all the time. Without touch typing, it's, for example, left index + right index. Or something else. Point is, you are not making the same moves all the time.
If you do, you develop RSI. For me, ALT + TAB was the problem. After ~15 years of touch typing, I started feeling first discomfort, and then pain in the wrist. Took some time to discover what's the problem, so now I'm aware and taking action, exercise, stretching, changing layout, switching to split keyboard, etc.
Touch typing is good, but be careful. Make breaks regularly, stretch your hands, invest in the ergo equipment. Think upfront, react before the problems occur. Better safe than sorry.
Beyond typing, it applies when considering any specific position or movement to be the only ‘correct’ form and optimize to do it again and again in that exact specific way.
However correct it is supposed to be, that’s a recipe for RSI. One way to mitigate that is to take frequent breaks, other ways include deoptimizing your movements to have more variation, rotating input methods, use a different hand etc.
Basically, extreme optimization and sticking to correctness is in my opinion overrated.
I fully agree. I type ~90 wpm. This isn't that fast, but it's great for what I do. My only training is years of sitting at my desk and using my keyboard. I don't touch type, so when I get a new keyboard, it's a pain at first to get my muscle memory down (65-70 WPM maybe for 1-2 weeks), but then I'm back to 90 WPM and typing buttery smooth.
That’s interesting. Do you notice any meaningful loss of productivity during the 65WPM time compared to the 90WPM time?
I just did a word count on some emails I write and they’re usually about 100 words, but I know they took about 10 minutes to write. I have to think about it as I compose it, then edit for clarity. Obviously if I was too slow this time would go up but I’m wondering where the point of diminishing returns is.
No, I'm pretty good at expressing myself through text. My common phrases that I use in sentences and words that I use often are burned into my muscle memory, so I usually have a pretty good flow. I never considered how much time I might be wasting during those slower periods, but I'll probably think about it in the future.
MacOS’ cmd is a thumb press for me, except “cmd+space” which I just reposition my whole hand for, and making caps lock a control (a setting available in the system settings, not an addon or command line tweak or anything) means no more scrunching my wrist down and left to reach that. I miss the cmd thing very much when I’m not on Mac, and I did 15ish years of Windows and Linux first so it’s not because that’s what I knew first.
OTOH by making trackpads not-hell to use they’ve allowed me to discover that too many days in a row of that wreck my right wrist pretty badly.
The macOS program Karabiner (especially in combination with BetterTouchTools) lets you go far beyond this, mapping arbitrary keys not just to different modifier keys but to any action desired.
Incredibly useful for coding (I have Caps Lock change focus between my IDE and most recent non-IDE window for mouse-free iteration cycles) and Mac gaming (tab-targeting is much more powerful with the caps lock button in play!)
+1 on karabiner and BTT; if using karabiner, perhaps check out goku: it gives you a much nicer language for writing karabiner profiles than just slogging them out in pure json.
I think it's important to develop one's own style to fit one's needs. Touch typing isn't a system to follow -- it's a goal to achieve. I touch-type, but for example, I always use left shift and never use right shift. And when I do hold left shift, the other fingers on my left hand shift one key to the left (i.e. index on D, ring on A). And I'm someone who has done past work in typesetting and writing legal briefs, so I ain't no slouch. Maybe there's some theoretical inefficiency I could choose to iron out, but I haven't yet been hindered by my typing speed.
More people need to realise this and be cautious of such fads. I got caught up into the whole touch-typing hype and continued typing in that awkward posture for 3 months straight more than 12 hours a day and didn't stop even when my hands, shoulders and neck started hurting due to the awkward posture that's caused by touch-typing.
I am in the worst pain of my life right now. Absolutely regret it. Sure the error rate is slightly higher and you type slower. Atleast you won't be f*cked with RSI.
I’m really confused by this whole thread and just want to clarify. “Touch typing” just means typing “correctly” right? As in with your fingers on the home row and reaching to type fluently?
Honest question, how is it that there’s debate about this and comments from people (on Hacker News in particular) about intentionally learning it recently? Maybe I just haven’t noticed, but I feel like everyone I know below a certain (not even very young) age knows how to touch type, even if they’re not particularly tech savvy.
Regarding this parent in particular, are you saying that hunting and pecking is better and typing from the home row is a fad?
> I’m really confused by this whole thread and just want to clarify. “Touch typing” just means typing “correctly” right? As in with your fingers on the home row and reaching to type fluently?
Yeah, "touch-typing" is incorrectly used to refer to home-row touch-typing by people who hadn't learned another style.
I use a totally different one, with my left hand resting on roughly shift-a-w-d and my right on j-i-o-; which requires little to no twisting to reach every key. I think I'm also more likely to lift my hands than home-row typists I've seen - those keys are just resting positions between typing bursts, not actually where I move my fingers back to while typing. The whole thing is based on the edges of the keyboard, rather than the nubs on the f and j keys.
I think there's a spectrum between grandma-style hunt and peck typing and perfect touchtyping, and I think many of us are somewhere in between. If you haven't specifically learned 10-finger touch typing, that's probably not what you're doing.
My typing certainly isn't Correct(TM), but some combination of muscle memory and bad habits formed over 30 years.
Yeah like what @encom mentioned below. I don't know about others but for me* it was a complete disaster. Your hands are always in one place.
The posture is very similar to the posture that is required when lifting a single dumbbell with both hands. Hands are side-by-side and your hands sort of form a triangle. This posture also causes shoulders to hunch forward to make up for the hands stuck close together.
Sure. It might work really well for everyone. Not for me. I am going back to normal typing once I heal. It wasn't really that bad. Just a few errors here and there and slightly slower typing.
There is touch typing and correct touch typing. "Home row" and friends is the latter. I'm typing pretty fast with all ten fingers in both German and Russian layouts, however, I've never became friends with classical touch typing. This circumstance make using split keyboards impossible for me, since I tend to cover more of the keyboard with my right hand than considered "correct".
I don't understand how touch-typing forces an awkward posture. I had a self-taught typing style for something like 10 years before switching to touch-typing about 4 years ago. I'm just as capable of slouching, leaning, raising, and lowering my posture like I did before. I started getting pain in my shoulders and neck but that was from pushing my hands together on a tiny keyboard. As soon as I switched to a split keyboard, all of that went away.
Just because someone says 'you will get RST when you do x' doesn't mean its true.
I'm touch typing now for over 18 years, quite fast, basically started with touch typing and i don't have issues.
There are plenty of alternative keyboards out there which will allow you to keep touch typing = fast and more ergonomic.
Alone the fact, that you need to look down to find your keys is weird to me. How do you correct your text while typing? Looking up and down all the time?
Personally I never have any issue with using keyboard. It's the excessive usage of mouse that is detrimental for my (right) hand (not just the wrist, the fingers hurt more from clicking and using scroll wheel too much.)
Not related to touch typing but the same is true of the eyes, focusing at a fixed distance all day kills the muscles.
I would guess that's often the issue with RSI as well, that the muscles don't get enough variation. I've practiced a lot of martial arts, yoga and rock climbing along the way and never had any problems despite spending A LOT of time with keyboards.
I've been touch typing naturally for way over a decade without paying much attention to technique. I have never felt (or paid attention to?) discomfort before reading about RSIs on HN. After looking into the lack of science around it and reading Dr. Sarno's book I have decided it's not something that's worth caring or thinking about. And perhaps even a dangerous idea.
I feel terrible for everyone experiencing this pain, but I'm not convinced it's something that can be reliably developed or avoided.
Obviously if you're feeling pain or discomfort for your own sake take a break or switch to a different finger for a few minutes. Just like you'd switch sides in bed or change your sitting style when feeling discomfort. But IMO this obsessing about developing an RSI is unhealthy.
You are unconvinced until it hits you. I never heard about it before it started bothering me.
It's individual. I have a friend, overweight, never played any sport, never saw him run, sits 16h a day, sleeps 8h a day, has no problem with his back. Another one, skinny, does sports, sits on a pilates ball at work, uses standing table, stretching, and still has back problems.
You can't make conclusions based on your own case. We are all different people, different physiology. Medicine is not maths, 2 + 2 is not always 4. The fact that they can't explain RSI doesn't mean it doesn't exist. It's just that we don't fully understand what's going on there (yet).
Right which is why you can read about physically unexplainable chronic pain conditions at large. It doesn't mean that people don't have RSIs, TMJ or whatever. There's simply no evidence that these are reliably linked to some specific physical aliment. There is no reason to worry about developing these things. That you feel discomfort in your fingers doens't mean you're developing an RSI. Just like a little disordered thinking doesn't mean you're developing schizophrenia.
Just the sheer understanding that fingers/wrists of an RSI sufferer are no different from these of non-sufferers helps many people resolve or greatly reduce their pain.
> Another one, skinny, does sports, sits on a pilates ball at work, uses standing table, stretching, and still has back problems.
And if they are a chronic pain sufferer from a similarly weird (or untreatable) condition I would greatly recommend looking beyond the physical explanation of their pain.
I think hunt-and-peck is still a repetitive motion that can cause problems of its own. I think it's better to take care of your hands with strengthening exercises and the like (I got some from an OT and they helped) than to just rely on not having proper technique.
I've heard one is supposed to use the palm to press Ctrl and not pinky. Though IME the palm is awkward and difficult to use precisely. Maybe it gets better with practice.
Swapping control and caps is another alternative, though in that case it's still using the pinky.
I believe that one of the original best practices of touch typing is to use the opposite hand to press any modifier key. This allows one to shift the hand so that the strain through the digit does not lead to an eventual injury. Given the number of strain complaints people that revolve around command key combinations, it would seem to be right up there with keeping your wrists floated in terms of injury prevention.
No downtime is acceptable, but they have only one server?
What if a technical failure happen? What if there's a fire in the server room? What if there is an earthquake and the building collapses? What if... many things can happen that can result in a long, long downtime with this tactics.
If uptime is so crucial, the system should be setup in such way that moving one server should be a peace of cake, not a spec-ops mission.
You’d be shocked how rare downtime is with modern hardware. A redundant power supply and SSDs in the right RAID configuration typically will not have any issues for years until it can be replaced by a newer model. Also, hardware monitoring is significantly improved to the point where you’ll typically know if something will fail and can schedule the maintenance.
In the past power supplies and spinning disc hard drives would fail much more often.
It’s basically a solved problem, outside of extremely mission critical, 5 nines kind of stuff, that we all forgot because of AWS.
HN ran, and may still run, on a single bare metal server.
> HN ran, and may still run, on a single bare metal server.
I bet HN wouldn't do a 10 hours high-risk operation for moving their servers because they can't afford an outage. (But well, running stuff on a single bare-metal server is expensive enough that even if they could, I expect they don't.)
What would that company do if a pipe broke inside the datacenter? Besides, if you never restart your servers, you are guaranteeing that the one time when the power goes off on the entire city, they won't come back online.
The point is that they probably also wouldn't then insist on a consultant doing an unreasonable migration and threatening to not pay them if there was downtime. And they probably wouldn't call around to other consultants with the same requirements, apparently telling them that the first consultant refused to do the job.
> apparently telling them that the first consultant refused to do the job.
While I don’t think they informed them of this in good-faith, it is a nice heads-up. In this case, it meant Consultant2 consulting RefusingConsultant that probably knew the IT better.
Quality hardware has existed for years. At a ford motor plant they were doing an inventory and couldn't locate a 10 ton mainframe. It was working so well for 15 or so years the tribal knowledge of where it was physically located was lost.
Wow, that's impressive losing that big a piece of hardware.
Though it was likely easier to find than that Novell Netware server that was sealed behind some drywall, with only a stray network cable leaving any clue as to where it was.
Depends on how big the building is that houses it – manufacturing IT can deal with impressive floor spaces.
I once only half jokingly suggested finding a missing data closet in a two million square foot distribution center by pinging a known IP from three or four aggregator switches across the building and triangulating the location on a floor plan. Sadly the people crawling around the ceiling found it before I could put my idea into practice.
2Msqft is c.430m x 430m for a square floorplan. Ping resolution is 1us (microsecond). Speed of electrical signal in cooper is about 0.8c. Gives a max resolution of ~240m by my reckoning. If there are variances in the switch+network delay it seems like you're going to struggle to even say which side of the building it is.
Hah! Good math. Based on the switch placement and the building being more of a rectangle I figured "north side or south side" would be as close as I could get. And when we really dug in it was a classic last mile problem: the first several core switches were well known, we just needed to figure out where the last aggregate switch went.
Turns out a door was closed and a new one built to a hallway to another hallway and not properly labeled on the updated drawings. Had one of the boxes running a conveyor belt not have died, we'd never have looked.
Often these kinds of SLAs are decided upon based on blame rather than what is reasonably required by the customers of that system. In this case, moving offices means the downtime is due to internal reasons. But if an ISP goes down or there is a natural disaster, then that isn't in their control.
Also cost does come in play as well. Multiple physical links in would be very expensive for what sounds like internal services. Likewise a natural disaster might cause bigger issues to the company than those internal services going down. They might still have offsite back ups (I'd hope they would!) so at least they can recover the services but the cost of having a live redundancy system off site might not justify those risk factors.
The customers requires are definitely unreasonable though. I'd hope those systems are regularly patched, in which case when is downtime for that scheduled and why is that acceptable but not when you're physically moving the server? I doesn't really make much sense; but then "not making much sense" also quite a common problem when providing IT services for others.
You are right, their SLA can be a bit different from what we're talking about here (and expect).
In general, we don't know much about this case. It's a post on Reddit, might not even be true. As is, it doesn't make much sense, but we don't know all the details, so maybe we jumped to conclusions.
Still, sooner or later, the data center will be hit by a natural disaster, a DoS attack, a network problem, or the like, and you'll have to be ready to move to a different one to get your service back online. Or you'll have to reboot your server to apply a critical kernel security update, in which case you need to be ready to fail over to a hot standby. So, since relying on a single server with high-uptime hardware is penny-smart and pound-foolish, might as go with a cloud-style architecture with commodity hardware.
I use to be fascinated with datacenters and would masquerade as a customer prospect to get a tour and see all the cool gear. I was asking one engineer about what they're plan was for a tornado (this was at ThePlanet in Dallas TX way back when) and they basically scoffed at the question. A week or so later one briefly touched down about 1/4 mile from them, I wonder if they thought about me when the sirens were going off hah.
AWS and older hardware is no different. Set it once and it keeps running for many years.
I've came across old AWS account (startup have been using AWS for the longest). All the network traffic or VPN goes through a single instance with 3 years of uptime.
True fact! I recently had EC2 migrate my VM when the physical server it was on reached EOL. If they had fired my VM up again, I wouldn't have even noticed. They didn't. Fortunately it had an EBS volume and I was able to manually restart it without data loss.
Human error is a bigger cause of downtime than technical failure or natural disasters. And in practice, a single server like this tends to be a hand managed one-off which only exasperates the human error component.
Unfortunately complacency about how reliable modern hardware is can lead to neglecting things like off site backups. And other issues. Yeah your one big critical on premises server may be super reliable. But what happens when the building is flooded with 6 ft of water, catches on fire, is leveled in an earthquake, or anything else?
If a function is super critical to business, it also deserves to have some thought put into the blast radius of its failure.
The sort of places that would insist on rolling a live server 700 ft across a parking lot probably don't have any real disaster recovery plan.
And multiple power supplies. I have been running a single physical server like this for ~10 years and the only downtimes were me restarting to boot a new kernel and when people at datacenter messed up BGP routing (their fault). HW is really very reliable now, especially in datacenter environment. But still not 100% of course. There is still low, but more lower than most think, probability of it failing. IC chips most likely won't break, only some capacitors degrade over time and flash memories with bios normally guarantee only 10 years. Bios upgrade (new write) would prolong that, though. I had one disk fail in RAID. Changed the drive without any downtime.
ECC for RAM is the other big one. A single-bit error will trigger warnings, so that you can replace the faulty DIMM before it progresses into uncorrectable errors.
Which is why AWS instances should be no more than minions in a load balancer pool, and any permanent state on an EBS volume or a managed storage service.
From an ISP perspective this seems like the sort of company that orders one $250 a month business DIA circuit (at a price point where there is no ISP ROI for building a true ring topology to feed a stub customer) and has no backup circuit. Then the inevitable happens like a dump truck 2km away with a raised dump driving through aerial fiber and causing an 18 hour outage.
Some circuits might average 5 to 7 nines of uptime over a year, but the next year is dump truck time... You can never truly be certain.
I worked at my last job for a place with a single rack mounted set of Windows servers at a data center - with no backup power supply, no backups of any kind for that matter, no UPS and no redundancy of any system, plus they didn't even have an admin for 6 months. The CEO refused to spend money on a 2nd anything. The company has 2000 employees. One server held all of the companies photos (which is basically the core of the business) and of course was not backed up.
You wrote one server but describe the failure modes of having one data center. I think it is very very uncommon and hard to allow for data center level issue. After all Instagram and 100 other site failed when one AWS data center went down. I would interested to know how/whether anyone's backend will work if any data center and its databases completely fails due to fire/earthquake/networking etc.
Second thing is having multiple machines for server. In theory it might help in increasing the availability but in practice I haven't seen any random issue due to machine which occurs just based on probability. I think almost all failure modes that exist, they are correlated between machines. eg suppose you have data loss on one machine, you could more likely than not, blame it on code and it would be similar across machines.
Re: single datacenter. At the basic level, you need a second datacenter with enough machines to provide your service (or a emergency version at least), replication of data, and a way to switch traffic. It's doable, but expensive in capital and development. If you're dependant on outsourced services, they also need to be available from both datacenters and not served from only one. In an ideal world, your two datacenters would be managed by different companies, so you would avoid any one company's global routing failure (IBM had one recently).
Re: multiple servers. Power supplies fail, memory modules fail, cpus fail, fans fail, storage drives fail. Sometimes those are correlated --- the HP SSDs that failed when the power on hours hit a limit (two separate models) are going to be pretty correlated if they were purchased new and stuck into servers at a similar time and then on 24/7. Most of those failures aren't that correlated though. Software failures would be more likely to be correlated though, of course.
The key thing is to really think about what the cost for being down is, how long is acceptable/desirable to be down, and how much you're willing to spend to hit those goals.
> In an ideal world, your two datacenters would be managed by different companies, so you would avoid any one company's global routing failure
I can't understand this. I think transferring servers would be the the least of problems. Its the transferring of database and maintaining consistent version of databases in both the locations. Moving the snapshots after every X minutes doesn't maintain consistency. I would like to read about any company that is able to do this, as honestly it sounds really hard to me. Is there any writeup of IBM thing you mentioned?
TLDR is connectivity to and from the IBM cloud datacenters (which includes softlayer) was generally unavailable, globally, for a couple hours. If you were in multiple IBM datacenters, you were as down as if you were in only one (mostly, I was poking around when it was wrapping up, and some datacenters came back earlier than others).
> Its the transferring of database and maintaining consistent version of databases in both the locations. Moving the snapshots after every X minutes doesn't maintain consistency. I would like to read about any company that is able to do this, as honestly it sounds really hard to me
The gold standard here is two-phase commit. Of course, that subjects every transaction to delay, so people tend not to do that. The close enough version is MySQL (or other DB) replication, monitor that the replication stream is pretty current and hope not a lot is lost when a datacenter dies. There's room to fiddle with failover and reconciliation; I recommend against automatic failover for writes, because it gets really messy if you get a split brain situation --- some of your hosts see one write server available and others see another, and you may accept conflicting writes. A few minutes running like that can mean days or weeks of reconciliation, if you didn't build for reconciliation.
He should have taken it offline without notifying this brain-dead manager. Probably wouldn't have noticed lol.
And then charge for those 5 hours for good measure.
In general, this stupid trend of wanting 0 downtime makes no sense to me. If you're not NASA, police or other emergency service you 100% can afford a few hours of downtime with scheduling it be forehead.
We used to have one server for a website I was a content guy on - it was in a standard PC case, plugged into a switch in the IT team's office (this was not a tech-centered org).
The main IT guy went on holiday and one of the cover guys from another office decided to tidy up. He unplugged the server and thought (and told me after his thought process) "if anyone was using it, they'll let us know".
This was the one, single box for the whole website - no one else was monitoring (even though the central office had a proper, dedicated web team) and the assumption was I was sysadmin.
An hour later I'm sprinting down the corridor to find out what the hell happened and why I can't even SSH into the box.
We put a sticker on the case saying not to unplug it after that...
IBM Mainframes are designed to be serviced while running so if you have multiple CPUs you can offline one at a time for upgrade it without the whole mainframe going down. Big Sun Solaris boxes where built like at as well.
If your mainframe had only one CPU, you did have to turn it off in order to service it. But you could upgrade the OS without turning it off. While they aren't cool tech now, mainframes are a marvel of hardware engineering.
