I was in this situation from the summer of 2008 until about a month ago. I'm just now starting to feel the relief of having another technical person to bounce ideas off, and eventually, to be able to take time off without worrying that the servers could go down at any minute.
Being the one person responsible 24/7 for a web app that can not go down is incredibly stressful. Doing that year after year is no joke. I think it changes your life as dramatically as having kids does. I'll never look at technical decisions the same way again, even though I know things will be less stressful in the future.
There were many low points and I know if I didn't have cofounders and customers depending on me, I likely would have quit and found a "normal" job several times over the years.
But I can't even describe how satisfying it is to be able to make the lives of customers just a little bit better every time I check something off my todo list. I'm not curing cancer, but I also have a much closer connection to the work I do and the impact it has on the lives of real people than I ever have before. And our customers are amazing and very appreciative of the work we do. We've had many tell us that they'd quit their businesses if they had to go back to the days before they had our product. That makes all the stress and many, but not quite all, of the sacrifices worth it.
If you can, definitely find another technical cofounder from day one. But the reality is that many businesses just can't make that work without supplementing revenue with consulting or from some other source.
Congrats and good luck Garrett! Thanks for sharing your story.
>Being the one person responsible 24/7 for a web app that can not go down is incredibly stressful. Doing that year after year is no joke. I think it changes your life as dramatically as having kids does. I'll never look at technical decisions the same way again, even though I know things will be less stressful in the future.
I think you might be exaggerating about the kids, (I mean, I don't have any of my own, but I'm the oldest of 6... really quite a bit older than all but one of them, so I think I have some idea as to what is involved.) but I really agree with the rest of that. Everyone that makes technical decisions should have to take a turn on the pager.
I've been a parent since I was 18 and I have 4 kids. I don't say that lightly.
I've also been on pager duty before. But being the only person responsible for keeping a critical system running for years on end is an order of magnitude or two worse than standard page duty I've ever done. I should add, our business is very seasonal and very unpredictable.
> I've been a parent since I was 18 and I have 4 kids. I don't say that lightly.
Now, /that/ is impressive. I mean, yeah, I'm not impressed by the pager. I've been on pager more often than not from 17 onward. But, I don't have kids, and my significant other is also technical, also works a lot and is super understanding. (well, and it's really only been the last five years or so that I had a relationship I cared about more than my servers, anyhow. Maybe less than that. I like my servers a lot. and I think there is a trial period for partners.) - I mean, that's the hard part about pager for most people; "hey honey, I've gotta stop paying attention to you, bring out the laptop and fix something real quick" and that isn't all that much of a problem for me. I've never really been a 'planner' so getting high priority interrupts isn't a huge problem.
but yeah. juggling that /and/ kids would be tough. well, kids in general would be tough.
eh, have you ever been the only technical guy at a company? I think it's a pretty similar experience. Either way; being a sole founder, and being the only tech guy at a company is way easier (assuming you have the skills and power to fix the system so that you don't get paged more than once a week) than being on pager on a giant and seriously broken cluster for other people.
I've worked other places where the pager went off three times an hour. Sure, you only had one week out of four, and only 12 hours a day, but if you fucked up even a little bit, the company would lose hundreds of thousands of dollars. It was a completely insane situation, and completely broken, as it should have been pretty easy to automate most of what we did. (most of what we did was telling the cluster 'okay, don't use that node, it's broken') Now, why didn't we fix it? I dono. there were some sharp folks there. Maybe we were all too burnt out from pager? maybe it was internal resistance to just throwing nodes back to our (really shitty) burn-in routine? (I mean, the idea was you were supposed to log in to the broken node and troubleshoot rather than just pulling it. But realistically, you only did that for the first day or two.) In the end, most of us ended up leaving.
This was the only pager that has phased me. Out of being on pager more often than not more than 20 years (well, uh, I guess it was closer to 15 years at that time) that was the only job where the pager broke me. Really, that was the only job where the pager bothered me much at all.
But yeah, compared to that? being a single founder is super easy. I get to make all the decisions, so I can choose things that break less. I can take all the shortcuts I like, and avoid the shortcuts I don't like. If the system is unreliable, it's completely my fault. If I want to take a pay cut so that I can hire someone else to lighten the load? my decision.
I mean, it's kinda like being the only tech guy at a small company, but you have a whole heck of a lot more power, so you can make decisions like hiring a customer out of IRC to help you out, which the boss almost never approves.
>I've also been on pager duty before. But being the only person responsible for keeping a critical system running for years on end is an order of magnitude or two worse than standard page duty I've ever done. I should add, our business is very seasonal and very unpredictable.
I guess my main point is that normal pager (when you are the only/head sysadmin) has all the responsibilities of being the single founder on pager, with a whole lot less power to fix the root cause of those pages. Being the new guy in a large organization usually gets you a little bit less responsibility, (not all that much less if you've sold yourself as some kind of super-expert sysadmin, or at least, that's how I felt.) but very little power to actually fix root causes.
Personally, I think the power to fix the root cause makes the job incredibly easier, and incredibly more interesting and satisfying. Otherwise it feels like you are just playing whack-a-mole with the problems. Nothing bothers me quite as much as fixing a problem without knowing how I fixed it.
> I mean, yeah, I'm not impressed by the pager. I've been on pager more often than not from 17 onward. [...] Out of being on pager more often than not more than 20 years [...] that was the only job where the pager bothered me much at all.
So how does pager duty work with your employers? Most places I've worked have asked for a 15 minute response time (which effectively means being within ~12 minutes of home) and being sober. That means no going for a bicycle ride or a run; not going on dates, to restaurants, the theatre or cinema; not visiting friends or relatives; and not going to the gym or going swimming or going out dancing.
Doesn't pager duty basically mean you have to sit at home on the internet for all your free time?
My pager agreement is response in under 30 minutes (whether it's log-on or escalation calls), within 1 hour physically of work, and sober. There's no cellular modem, and you have to use the corp laptop to access the VPN to access your machine.
I asked my co-workers on rotation if they were interested in trying to get a cellular modem that we pass around in addition to the pager, but they would rather try to use iPass to get on the wifi somewhere.
Fifteen minutes is... kinda a long time for logging in and checking things out. The thing you miss, though, is that there are all sorts of devices with which I can log in.
I have ssh on my cellphone which is on my person at all times, and while doing anything involved is difficult, I can usually verify that there is a problem and it's nature, and if it's easy, I can deal with it then. But that satisfies the 'respond in five minutes' or whatever. I responded, I can tell people involved what's up and how long it's going to take me to get more info.
The next step up is my laptop, which usually stays in the car. I jog to the laptop, boot it up, use the cellular modem dongle thing, and it's about as good as being at home, most places. I have a cheap inverter in the car if my batteries are low.
If I'm willing to take a social hit, I can also call someone else and say "Hey, I'm drunk to the point where I can't handle root, or I'm 20 minutes out and my laptop took a shit. X paged me. can you deal with it?" and repeat until someone says yes. I look like a fool in front of my co-workers, but eh, the problem gets solved, and everyone does this every now and again.
(This is the big problem with being a sole founder. You can't just call up a random consultant; you need to know the system to be able to fix it. It's really nice to be able to call two or three people that know the system well enough to help you out in these times. In this case, I think even someone that doesn't have a lot of experience but is willing to be trained, and that you can afford to pay often enough that your system is fresh in their mind, is better than just calling someone really good that doesn't know the system when the system is hoarked.)
So yeah, I can go to movies. I might need to walk out of the movie, but I can go. Restaurants are easy; I don't even have to leave; I just whip out my laptop and fix what needs to be fixed. dancing? same deal. Yeah, I look like a dork or an asshole if I get paged and I whip out the laptop in the middle of a social engagement, but honestly? I'm not sure that really changes anything.
Really, anywhere I can take my laptop that has cell reception and power, I can go. I mean, I know I might get interrupted, but I probably won't be.
The hard part is really the mental state. You really want to make sure you go to bed before you are exhausted, because problems that take five minutes when you are fresh can take, well, much longer when you are not. Alcohol too; I find that the DMV limits are reasonable, and if I stay under those I can handle myself (on a computer; I'm a little more conservative when it comes to driving, as I'm a pretty shitty driver to begin with.) Sleep, really, is the problem I struggle with, but I have a hard time keeping a schedule anyhow.
(Oh, also, the 20 years is clearly a lie. I'm what, 31? something like that. and I think I got my first pager at 17, so considerably less than 20 years. For some reason, this week my sleep schedule has been pretty messed up, and I haven't been mentally 100%. I've been writing a lot, and writing poorly about stupid bullshit that doesn't matter. Odd.)
The life of a hosting company owner in a few brief paragraphs. "Me too" would be fairly easy to state, albeit slightly different: I had my operation when I met my significant other and as a result have a fairly varied and flexible schedule.
Overall I think she enjoys that, adding a child to the mix has been fun & surprisingly hasn't changed much at all -- in the first few months it gave me an even better excuse to be up at 4 in the morning.
I've been at this for about ~6 years as a solo guy (albeit, with hires -- so that changes things entirely) and a few years prior at other companies in slightly lesser capacities as an admin.
Which reminds me, I need to get back to you one of these days.
Being the one person responsible 24/7 for a web app that can not go down is incredibly stressful. Doing that year after year is no joke. I think it changes your life as dramatically as having kids does. I'll never look at technical decisions the same way again, even though I know things will be less stressful in the future.
There were many low points and I know if I didn't have cofounders and customers depending on me, I likely would have quit and found a "normal" job several times over the years.
But I can't even describe how satisfying it is to be able to make the lives of customers just a little bit better every time I check something off my todo list. I'm not curing cancer, but I also have a much closer connection to the work I do and the impact it has on the lives of real people than I ever have before. And our customers are amazing and very appreciative of the work we do. We've had many tell us that they'd quit their businesses if they had to go back to the days before they had our product. That makes all the stress and many, but not quite all, of the sacrifices worth it.
If you can, definitely find another technical cofounder from day one. But the reality is that many businesses just can't make that work without supplementing revenue with consulting or from some other source.
Congrats and good luck Garrett! Thanks for sharing your story.