> Nick was hired out of his job at Amazon because he was supposed to be the AWS expert.
This always cracked me up. From what I can tell, he was a mid level dev on the Alexa web api team. He knew AWS sure, but he did not have the cred at all to justify the position and responsibility he was given at Ubiquiti.
Hoo boy, this is gonna be a fun one. For reference, I spent a year (mid-2018 to mid-2019) running the UniFi Network team and worked with Nick during that time.
> * Why was it so easy for a lead engineer to get access to a root AWS user without anyone else being notified? I.e. AWS GuardDuty provides FREE alerting for when an AWS root IAM account is logged in or used, this account should be under lock and key and when used, confirmed and audited by relevant persons or teams.
The "Cloud Lead" that Nick took over from gave zero fucks. He ran all the AWS stuff for Ubiquiti under his personal AWS account. Nick came in and started putting "proper" AWS structure and security in place, primarily by scaring Robert (the CEO) into giving him the keys to the castle (my own personal opinion of Robert is... not the greatest).
One thing to understand about Ubiquiti (at least during those times) is that the company had zero C-level execs. There was Robert.... and then nobody knows. I asked repeatedly why we didn't have a CTO, or a COO, or a CFO, or CMO or ANYTHING and I got nothing but shrugs and "idunno" as a response for the whole year I was there.
So when Nick came in, a very... let's just say "forceful" personality, he immediately won over Robert and ended up with carte blanche over pretty much all of Ubiquiti's cloud accounts. Which were basically... everything. All the UniFi Network services, UniFi Protect services, you name it. If it was connected to the cloud in any way, Nick had access to it.
So why wasn't anybody else notified? Simple. Because he was basically "god". If anybody was gonna be notified, it would've been Nick. He was the top of the totem pole company-wide when it came to AWS.
Also, for some perspective, at that time Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to. And they were in plain-text. So... yeah.
> * Furthermore on the root account being easily accessed, the root account in the companies I've worked at had MFA enabled, and the QR code is locked in a safe only accessible by two people agreeing it needs to be accessed in a break glass situation, where warranted.
See above for the quality of security processes and practices this company had in place.
> * Why was he also able to delete critical CloudTrail logs and reduce their retention to 1 day? I.e. These logs should be in a S3 bucket or other environment where such changes cannot be made. Alternatively, they should be shipped to a redundant service that manages this risk to prevent data deletion
See above. (re: "god") Nick answered only to Robert. And he'd already successfully hoodwinked him. He could do whatever he wanted. Eventually he fell from Robert's good graces, but seeing as Ubiquiti as a company didn't really have a ton of checks and balances, he kept his god-level access far longer than he should've.
> * Why did Ubiquti not announce they were compromised sooner? The hack started in early December, Ubiquiti noticed the compromise on Dec. 28. Ubiquiti told the market on January 11th. Is that a satisfactory turn around? Giving them some credit for the XMas break I'll say this partially understandable.
Simple. Fear of share price falling. I was constantly given this as a reason we couldn't be transparent. Not by Robert, nor where he could hear. But it was pretty much well known that the company kept shit quiet for fear of the share price dipping.
> All the AWS configuration I'm speaking of above, I would describe as Security 101.
To keep with the metaphor, Ubiquiti couldn't even get Pre-school level security in place, much less 101. I have no idea how something even more massive hasn't happened yet. Must be dumb luck.
Speaking of, by the time I left the company, the team that was handling the door entry-way systems (UniFi "Access" I guess) had been caught with numerous security issues, not the least of which was logging user credentials in plain text (not just storing, but logging, in response to authentication events). They were also based in China and subject to Chinese laws around government access, so take that how you will.
And that doesn't really even cover most of it. That year took a toll on my physical, mental, and emotional health, not to mention put a crazy strain on my marriage. I'd rather honestly forget it, but the schadenfreude of what's going on is too delicious to ignore.
>Also, for some perspective, at that time Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to. And they were in plain-text. So... yeah.
This is frankly worse than any of this other news. So there's essentially zero trust associated with the code signatures since any employee, past or present, can sign a payload. Wonderful.
I've since heard that the repo has been taken down and all the keys rotated, but just kinda makes you wonder how many APs and switches and cloud keys, etc are still out there using compromised keys.
Also, even though they may have had read access, not many knew it existed. But it wasn't super hard to find (I stumbled across it basically).
Oh and then there the whole metrics collection debacle, where the controller basically phoned home about the topology of every network that it managed. Even if you opted out. Opting out just meant they fuzzed your ID so any given record couldn't be linked back to PII. Which may or may not be legal, IANAL.
But either way it definitely wasn't clear that opting out meant data was still collected. Super sketchy.
> Also, even though they may have had read access, not many knew it existed. But it wasn't super hard to find (I stumbled across it basically).
We didn't have read access until Nick Sharp and his team took over GitHub permissions and gave everyone access. Wonderful security work.
> Oh and then there the whole metrics collection debacle, where the controller basically phoned home about the topology of every network that it managed. Even if you opted out. Opting out just meant they fuzzed your ID so any given record couldn't be linked back to PII. Which may or may not be legal, IANAL.
Nick Sharp was at the core of this too! He built the 'trace' system to collect all of these metrics and had all of these ideas about how to secretly collect the data in ways that would be hard for people to detect.
He pretended to be a principled person who stood for security and privacy, but whenever he saw an opportunity for political gain he abandoned all principles. He was the only person I knew at the company who was enthusiastic about collecting all of that data.
Oh god don't remind me about Trace. I had to deal with the Controller side of that and it was a damn nightmare.
He basically dictated that you couldn't use any kind or repo+deployment pipeline except for what his team was building. Which wasn't actually functional for like 8 months. So we never even got a dev or staging tier to test against for months.
And then when I ended up with access to push things along, the actual apps for the trace system we're... not well implemented.
Ugh... I could bitch about this stuff for literal days but I gotta drop my kids off.
Oh hai people who used to work at UBNT. From reading your responses here (and elsewhere) it definitely seems our paths did not cross, but the shared sense of Schadenfreude is good and strong :)
The usual answer I've found to this question is Microtik, they strike a similar balance between enterprise feature-set and more consumer level price point as Ubiquiti sort of aims for with the Unifi line.
The quality/feature set is there and the software is well designed, even if not quite as networking beginner-friendly as Unifi has become. Mikrotik's RouterOS can do much the same tasks as Unifi's management console, and can configure for auto-adoption of APs/other hardware in the Mikrotik range just like Unifi does for their own hardware.
Most competitors (I see Aruba suggested) are priced much more into the enterprise/business buyer realm. Unifi has generally been keenly priced in this market, their latest Wifi 6 APs are just 99 dollars each (when in stock of course...). Microtik's pricing is generally comparable or cheaper than Unifi in my experience.
Secondhand Ruckus APs are a pretty decent alternative, you'll have a hard time getting AX gear for a reasonable price though.
edit: Secondhand Ruckus/Brocade switches are solid, at least on the 7000 series the evaluation key has no time limit so you're not license-limited in what you can do with them. Switches are mostly <$250 on eBay if you're buying an ICX7150, ICX7250, etc. Yes, that includes PoE models.
Is it just software that's UniFi's weakness? Anything wrong with the hardware itself? I've had quite good luck with UniFi in my home myself but perhaps I'm not using all the features...
> Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to
This right here is why I'll never use Ubiquiti gear. These devices are so obviously backdoored and like swiss cheese, they offer the complete opposite of security. Thanks for sharing the true facts.
> For reference, I spent a year (mid-2018 to mid-2019) running the UniFi Network team and worked with Nick during that time.
Nick's whole strategy was to find a problem, exaggerate it as much as he could get away with, and then offer himself as the hero who would fix it all.
He exaggerated or lied about everything he wanted to use for political advantage, right up to the end where he fabricated a hack and used Krebs to exaggerate it as much as possible for his own personal profit.
You have to realize he did the same thing during his time at Ubiquiti: Found problems he could use for political advantage, exaggerated them as much as he could get away with, and then amplified his lies until they were gospel. A lot of what you're saying has some roots in truth, but I can tell you have the exaggerated Nick Sharp version of events.
> There was Robert.... and then nobody knows. I asked repeatedly why we didn't have a CTO, or a COO, or a CFO, or CMO or ANYTHING and I got nothing but shrugs and "idunno" as a response for the whole year I was there.
This wasn't some big mystery. Everyone knew that Robert ran everything as CEO and the legal, marketing, and other teams operated out of the New York office.
> Nick came in and started putting "proper" AWS structure and security in place, primarily by scaring Robert (the CEO) into giving him the keys to the castle
Nick was hired specifically to run AWS. That was his job from the beginning. The old cloud team quit and Nick was recruited from his job at Amazon because supposedly he was an AWS expert.
The incident where he scared the CEO was the first of his political games to exaggerate or fabricate security incidents for political gain.
> So why wasn't anybody else notified? Simple. Because he was basically "god". If anybody was gonna be notified, it would've been Nick. He was the top of the totem pole company-wide when it came to AWS.
Yes, this. All of these news stories are missing the point that Nick was the cloud lead. You don't have to believe anonymous commenters. His LinkedIn profile will confirm it. He was recruited out of Amazon to lead the cloud efforts, but he was in over his head and had severe personal issues.
> at that time Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to.
This is another Nick exaggeration. It's true that older devices had hardware signing keys stored in a Git repo before the system was updated and keys rotated. However, those old keys were only accessible by a few people until Nick and his team took over GitHub and restructured permissions with the web portal they built themselves. In the process they made too many repos accessible to too many people.
> To keep with the metaphor, Ubiquiti couldn't even get Pre-school level security in place, much less 101. I have no idea how something even more massive hasn't happened yet. Must be dumb luck.
Ubiquiti's overall structure is far from perfect, but you were only there during the Nick Sharp era. Ubiquiti had a lot of people who took security and proper practices very seriously before Nick Sharp took over everything, but it was also a distributed company with a lot of isolated divisions. Nick Sharp got into power by taking the worst and oldest parts of the company and convincing people that everything was equally bad and that only he could fix it. If you got your security information from Nick Sharp, you'd think that Nick is the only person who can do anything properly at the company.
> Speaking of, by the time I left the company, the team that was handling the door entry-way systems (UniFi "Access" I guess) had been caught with numerous security issues, not the least of which was logging user credentials in plain text (not just storing, but logging, in response to authentication events). They were also based in China and subject to Chinese laws around government access, so take that how you will.
I also heard that, but I think it was just incompetence on their part. Nick was pushing the conspiracy that they were doing something with the Chinese government, but it doesn't follow that they'd do it by sending the data to AWS servers under his control. I think they just made a sloppy prototype to impress the CEO and got caught doing dumb stuff. I do blame the company for not cutting that team off, though. They had no idea what they were doing other than their ability to put together quick prototypes to impress the CEO.
If you're telling me I worked there at literally the worst possible time frame, I'd believe it. I may have my experience skewed through the perspective of Nick's influence, but tbh many of my issues were unrelated to him or his sphere of influence.
The C level thing may not have been a "big" mystery, but it was to me, and as somebody who was running the dev of a flagship software product (UniFi) it set off alarm bells that nobody I talked to could explain who was handling the roles of those execs. I'm not exaggerating when I say I effectively got "I dunno" as a response when I inquired, and I dug.
It is good to know, though, that what I experienced wasn't chronic for the entire company's existence.
To clarify on the China thing, I wasn't trying to imply that anything nefarious was actually happening. Just that it warranted some scrutiny when a security focused product was being developed on the Chinese mainland and by a team of Chinese citizens that are subject to CCP laws. Given some of the things that have happened around that country's involvement in tech in recent years, I don't think such scrutiny is unwarranted, especially when the team has a track record of security "goofs".
Dude, let's not be generous. Could he write code? Yes. But this is a guy who wrote everything in Node, but absolutely _refused_ to use any existing libraries except for ones he personally wrote. He didn't "trust" them.
He wasn't even hired on as a dev, he was hired to be the "Cloud guy", essentially a sysadmin for AWS, and basically spooked the CEO into giving him the keys to the castle.
A create-react-app app is not a node app (It has a node dev server, but it's a front-end JS app), so its a weird thing to reach for to illustrate a point about node apps.
Picking a language (and possibly runtime) is a pretty huge investment if you intend to become proficient. A lot of people like to think that they are polyglot programmers and that language doesn't really matter. But it does. It takes a few years to become a decent programmer in a given language. And if people claim it takes just weeks or a couple of months, it really only tells you that they have very low standards.
If you are familiar with a given language, ecosystem and runtime, and you care about productivity and quality, the path of least resistance is to stick to what you know. Taking on a major project in a language you don't know is a risky proposition. In terms of quality, time, and even in terms of being able to deliver something acceptable.
I tend to have a main workhorse language. It typically takes 2-3 years to reach an acceptable level of comfortable familiarity with a new language. If history is any guide I tend to stick to the same language for 5-10 years. 5 years ago I switched from Java to Go. I mostly worked mostly as a manager at the time, which is why it took longer to reach what I think is an acceptable level. I'd say it is only in the last 18 months or so I've started feeling sufficiently competent in Go to call myself a Go programmer.
That being said: I think the JS space is both a poor technical choice and a poor career choice. The whole ecosystem is janky as fuck, you have to spend a lot of time dealing with silly complexity that tries to fix the jankiness, and the type of work you get isn't very attractive.
It takes years to become acquianted with the library ecosystem of a given language. If you're going to write everything from scratch (especially in a language with an extremely bare-bones standard library), it takes maybe months to become proficient with any language in a paradigm you already know, save a few extremes (C++).
If I can vent for a second, this company has no leadership. None. Things may have changed in 2 years, but I doubt it. I was messaged almost daily by random employees asking wtf was going on with the company. They were afraid for their jobs. Practically no one respected the CEO, and he was the only C-suite exec. There. Was. No. Leadership.
There was no company wide communication, and all communication channels were made private, and if you sent an email to more than a couple people you were directly rebuked by the CEO. Nobody felt like they were trusted, and the norm was for most engineers to have absolutely zero idea of what was happening in the company outside of their direct project.
Teams were constantly at odds and pitted against each other, and the CEO never resolved any conflicts between teams or employees. The company (at least the software side) was treated like Thunderdome. Some team leads and office managers took care of their people, but most people were just beaten down. I don't think I'd ever seen a less motivated, more dejected group of software developers than I did during my time there.
IMO, this kind of bullshit clown show starts from the top. And as long as the top doesn't want to fix it, it won't get fixed. And since software almost invariable ends up reflecting the structure of the organization that produced it, you get this kind of security shit show.
I hope this is the last one and they get their act together. But realistically I can't believe that'll happen.
> There was no company wide communication, and all communication channels were made private
I couldn't understand why the ex-Amazon cloud lead was also in charge of Slack. When he made all channels private and put a Slackbot in every channel to monitor conversations, I knew it was all over. I'm worried his Slackbot logs are part of the leak. Guy had his hands in everything :(
Same guy who took over GitHub and forced everyone into his self hosted source control because he couldn't trust Github. That decision didn't pay off.
> I couldn't understand why the ex-Amazon cloud lead was also in charge of Slack. When he made all channels private and put a Slackbot in every channel to monitor conversations, I knew it was all over. I'm worried his Slackbot logs are part of the leak. Guy had his hands in everything :(
He did.... what? That sounds like straight out of a Dilbert comic.
I mean, I didn't necessarily agree with all of his methods or reasonings on everything, but I've come to realize a lot of times his hands were just as tied as ours. And the draconian surveillance stuff? Yeah, he was directed to do that. One guess by whom.
He was "in charge" because he convinced Robert that he was the right guy for the job by finding a security flaw that let him log into Robert's personal UniFi Protect setup at his home. At that point Robert basically gave him carte blanche, but also started directing him to lock everything down. More than a bit of paranoia there, in my opinion.
He was in charge of cloud when he "found" a way to forge Ubiquiti SSO logins for any user using his root access to the SSO signing secrets.
In the Krebs article the whistleblower calls out forging SSO logins as one of the things that was compromised. If the attacker is really an ex-employee like Ubiquiti says, then it's scary that the SSO signing keys aren't even being rotated after the account forgery stunt.
> Adam says the attacker(s) had access to privileged credentials that were previously stored in the LastPass account of a Ubiquiti IT employee, and gained root administrator access to all Ubiquiti AWS accounts, including all S3 data buckets, all application logs, all databases, all user database credentials, and secrets required to forge single sign-on (SSO) cookies.
From the outside it seems like accepting fault and product returns would smooth waters. Acknowledge faults on their own forums and Reddit subs and also provide times lines for fixes (then stick to them and update threads!)
The hardware is mostly good. The weird bugs and company management are turning a strong community of users against Ubiquiti.
Even as an outsider it's beyond obvious there is no leadership or vision other than cut costs.
After Brandon left, Unifi went to shit. There hasn't been one significant feature or major function added to Unifi since then. Routing hasn't move at all in 7 years. Well, in a recent beta you can now have multiple WAN IP addresses. Whooppee. Switching hasn't gained anything - layer 3 is utter missing. QoS? Good luck.
Unifi is fine for networks with simple needs, good for prosumer use or small businesses - but if you start to scale requirements it falls over pretty quick.
It was very promising when routing/switching was added to Unifi - but it's never been fully realized :(
> I hope this is the last one and they get their act together. But realistically I can't believe that'll happen.
The good thing about them being a public company is there is some accountability from outside the company. Looks like they're already being investigated for fraud for downplaying the breach and their stock price took a big hit. Hopefully this all leads to the CEO being replaced and things turning around.
> Ubiquiti also hinted it had an idea of who was behind the attack, saying it has “well-developed evidence that the perpetrator is an individual with intricate knowledge of our cloud infrastructure. As we are cooperating with law enforcement in an ongoing investigation, we cannot comment further.”
I personally don't believe this. IMO, this is a company who is looking for a fall guy, and _most likely_ it's going to be somebody who raised a stink about all the security problems during their time there.
Form your own opinion, I'm just a guy who worked at Ubiquiti for a year, raising all kinds of hell about the security, architectural, and operational problems that I saw while I was there.
That would be the reverse of the usual strategy, wouldn't it? Most companies seem to try to pin breaches on sophisticated hacker groups backed by nation states. But then, they benefit from the perception of a threat that's impossible to defend from (so there wasn't anything they could do) - whereas Ubiquiti benefits from people thinking the attack was just a small actor that couldn't possibly threaten Ubiquiti's customers.
Yes, you're right. But I don't really expect them to make the "smart" or "usual" play. That would honestly surprise me. Now, pinning it on somebody that was generally disliked because they constantly blocked things that had obvious gaping security holes? Basically sicking law-enforcement on somebody out of pure spite? I can absolutely believe that.
Accusing whistleblowers and reporters is indeed common - it pretty much seems the standard behavior in infosec in particular.
What I meant was something different. The breach, as I understand it, was quite critical. Ubiquiti in this case could take the standard corporate spiel of "it has hallmarks of a nation state attack, there was nothing we could do" bullshit disclaimer - but given the nature of this breach, every customer of theirs would now be wondering if $Enemy has put malware in their infra, and whether it isn't a good idea to smash it all with a hammer and buy new one from someone else. So I suspect Ubiquiti is going the other way, blaming it on a single, inconsequential individual, that absolutely, positively didn't give access to anyone else, and thus nobody's infra was in any danger.
(Note: I have no inside knowledge, or even any deep knowledge, of this topic - I'm just a random Internet person speculating.)
Nah, most of the time it's just a fancy infosec way of saying "it was likely ordinary criminals, or even some script kiddies, but it would be quite embarrassing to admit that".
Thanks! Aha, so "state" here refers to the people running the country?
Rather than the citizens / "all people living there", (or geographical area)
So, could be organizations indirectly paid by, just an example, Putin or the CCP. But who the people in Russia or China don't know about (and might not have supported). -- I'm not a native speaker (I guess you've noticed :-))
I really wouldn't like to migrate away but I can't say all the info that's been coming back has been making me want to have them as a part of my network infrastructure.
I want to fire Ubiquiti, but where can I go to get my router, wireless access points and switches in one management interface? There are plenty of poorly performing consumer grade options out there which hide all complexity, but they break in fun ways (eg: Google WiFi creating loops in the network when users try to do wired backhaul) and only tackle part of the stack.
I really just want to manage an OpenWRT based network with one central web interface and not have to deal with corporate/state entities deciding to push fun changes out in the management interfaces that power these systems.
I keep seeing the requests for central management interface, which leave me somewhat puzzled. Why do you need in a home environment? I run a small network with one big router and several access points, and at least with Mikrotik's gear, it's pretty much fire and forget. It has CAPsMAN[1] to centrally manage wireless networks, but I've found it to introduce unneeded complexity. Auto-updates[2] don't need any central management either. Monitoring can be done through SNMP[3], and there's a REST API too[4].
I have a good deal of experience with Mikrotik's offerings, and I am not looking to power networks I support with a patchwork of different systems that each have their own interface.
Most of the value proposition of the Unifi lineup is I can look at a single website that I host and see the WiFi clients connected to an access point, what switch feeds that access point internet (and whether its linked at gigabit or 100Mbps), uptime on all devices involved in the stack, whether the client has poor WiFi quality, trouble DHCPing, etc.
The single pane of glass to view everything when I am many miles from the networks I support is essential. Compared to when these sites were on PFSense before migrating, these networks have improved uptime, rapid remediation of issues, and changing VLANs, SSIDs and labeling each client on the network is a snap.
Edit: Borrowed /u/bpye's single pane of glass term
It's definitely not all the new controllers, although with the UDM line you might be right. I think there's a huge intersection between people who would buy those specific devices and people who are perfectly happy to have remote access to their control plane in the cloud.
It is also about dark patterns. I never had the cloud option enabled. One night after a long day I upgraded the controller software. I noticed a message like “do you want to login?” and wasn’t awake enough to realise that it asked for my ui.com account and that after that cloud management was enabled and my phone switched to authenticate from a direct connection with the local credentials to using the ui.com credentials.
It looks like what I was referring to is that they recently made the initial controller setup on the cloudkey require a cloud account [1], but you can migrate to local only after the initial setup.
So the only remaining 'local only' from start to finish is for self-hosted I guess.
I have a cloud key gen2 plus and do not have a UI.com account. I would classify getting the network controller setup without having one initially "mildly annoying but worth it".
I'm also floored at the number of people who are spinning the existence of a self-hosted controller as somehow a bad thing...?
The UDM and UDM-Pro force you to set up a UI.com account, and cannot be used with external Unifi controllers like one you might run on a server, PC or cloud key (Ubiquiti's management software on a Power over Ethernet powered dongle, does not require a UI.com account).
They do - first thing I did though was then go in and add a local account, and disable remote access (I have a wireguard tunnel that terminates on a server behind my firewall if I need remote access).
I don't use a UI.com account to connect to the Unifi controller I host (as I don't need their inconsistently working NAT traversal to get to my controller), hopefully the networks I support are safe due to not being entangled with Ubiquiti's cloud infrastructure.
Anyone who is forced to get a UI.com account (eg: UniFi Dream Machine and UDM-Pro owners) should change their credentials and do a factory reset on their routers and Access Points ASAP.
> do a factory reset on their routers and Access Points ASAP
This is a miserable user experience. If you do a reset and don’t know the SSH password on APs or cameras you get to spend a hellish few hours crawling though ceiling insulation, climbing ladders and physically resetting devices. It’s so shit. I’ve just done it, but not due to security concerns, but instead because of a UDM-P crapping out randomly.
This is why I like having the controller in a virtual machine offsite. Factory resetting the router and pairing it to the same site in the separate controller gets me back to the same exact place I expect to be.
With the UDM series, the integrated controller ensures you lose everything if you have to factory reset, site to site VPNs have to be manually configured, and numerous other minor annoyances crop up (like UI.com not always being able to connect to the controller).
>If you do a reset and don’t know the SSH password on APs or cameras
Who's fault is that if you don't have it? First thing I do when I set a new site up is record all the vital information like that for when I will inevitably need to recover stuff.
It should be standard backup/disaster recovery practices - for ANY system. Making sure you have critical information BEFORE you really need it is preparedness 101.
Similar to the other responses, it's the fact that I can manage my network remotely from a simple app or UI. This helps me answer phone calls from my family asking why Netflix doesn't work on TV #2, when I'm not at home. Won't solve all problems, but at least I can narrow it down and troubleshoot.
And I like the fact that I can an overview of the state of my network; one of my wired links to an AP would degrade to 100 Mbps at times, and being able to see the link speeds easily was very helpful (it was a bad ethernet cable in the end).
Before I moved to Ubiquiti I had a spate of problems with my fiber broadband, which would stop working for a few minutes at random, resetting my RDP connections. I had a vendor-supplied Linksys (I think?) router, and trying to troubleshoot it was painful. If I ever have such problems again I'll have much better diagnostics.
That said, I won't buy any Ubiquiti gear that requires the cloud, and my faith in the company is eroding. But, like others, I would be at a loss what to replace my gear with at the moment. I just hope it'll function well enough until either Ubiquiti gets it act together (again?) or a viable competitor arises.
Network cables (copper and fibre) have a limited bend radius. Most people don't think about this and will bend a cable beyond tolerance, which will eventually result in the cable not working correctly, and/or manifest as intermittent issues.
I suspect that's the most common cause of network cables 'going bad' in the home.
I learned this back in school, when the previous years students had laid new Ethernet cables from the classroom to the server room, but the machines would only get 10M and not 100M link as they should.
Didn't take us long to notice they had laid the cable like electricians, neatly following the contours of a few door frames with tight 90 degree bends.
> I keep seeing the requests for central management interface, which leave me somewhat puzzled. Why do you need in a home environment?
Crap wifi was a huge thing I dealt with. Unifi fixed that completely. The ability to run a relatively complex network (by home network standards) with multi access points is nice, but the ability to administer them without CLI interface is great. I loved my edge router but touched it with trepidation. It was rock solid except when I was sucking with it. Unifi suits/suited the enthusiastic amateur.
> I run a small network with one big router and several access points, and at least with Mikrotik's gear, it's pretty much fire and forget.
Unifi used to be too, with an interface that was a bit difficult to navigate (settings spread among about 20 tabs, but it was possible to get the job done without sshing to components).
Now it’s flakey. I just rebuilt my last week which was working fine but I couldn’t log in and the UDM-P screen said it required resetting. Dark times.
To answer this for me personally (and I suspect this is a pretty common answer): To use the best, and to explore technologies that I might suggest to business clients.
Business clients love central management interfaces.
As well, I’m honestly kind of done with managing fiddly “snowflake” devices, and central management interfaces usually come with the ability to standardize the config across devices.
I definitely don't "need" it. But it's veeeeeeeery convenient. Especially when it comes to security, being able to see which devices have updates and perform them all from one screen, is extremely convenient. I'm highly interested in paying for convenience at home.
Thankfully I don't use their cloud based management interface -- as far as I know this breach does not affect my local UniFi Controller. Hopefully this is a rude awakening and Ubiquiti goes back to their old consumer focused approach.
Frankly I wonder at how big some of these peoples' houses are. My single seven year old Nighthawk router covers an entire 2300 square foot home and penetrates the brick walls to reach halfway up the street.
That’s not my experience, all the way from Meraki enterprise access points to the standard consumer WRT54GL.
First problem is 5GHz is terrible at going through walls, I don’t believe it will even go through a single brick wall and maintain decent bandwidth. Even 2.4GHz is considerably slowed by 2 or 3 drywall/plywood obstructions.
Second problem is can the mobile device you’re using return that signal through all those walls to the access point. I have noticed an huge increase in quality and snappiness of FaceTime and other high up and down bandwidth activities once I added more access points so that connections are going through only 2 or 3 walls.
For another reference, I have a hotel that needed to upgrade its network to meet the brand standards for signal strength in all the rooms, and we had to end up installing 6 access points in the drop ceiling of each hallway 15 guest rooms in length (each guest room is ~15ft wide, so the corridor was ~225ft long). It resulted in the elimination of almost all guest complaints about the wireless network.
Mine's only slightly larger than that (mostly by virtue of having 3.5 levels, not by X-Y size), but the original plaster walls attenuate the hell out of 5GHz signals. I have two APs, one in the basement and one on the second floor and even with that, I'm considering adding two more inside and a dedicated one outside to serve the patio/BBQ area as I can readily tell the speed difference to internal file and backup servers if I'm in the same room as an AP vs on another floor or outside.
Make no mistake, it still "works" with just one, only slower.
Somehow I have managed to spend most my time in a house that has concrete and brick stopping 5G, a house with wooden walls that block RF and foil insulation under the floor which is even worse, and a workplace environment that has literal faraday cages all around.
I like UniFi in wall access points in the room I’m inside.
My house is about that size. My detached garage is 400sqft. My barn is 1600 sqft. And my travel trailer is 37" long. My network comes into the house and the wireless needs to cover all of the structures because we need into in all the places. It's all spread over about an acre and a half. I run ethernet to a PoE AP in the garage, through an overhead crawl space that covers thale span between the house and the garage, I have b2b radios between the house and barn and the trailer has an LTE router/wifi repeater that picks up wireless from the barn.
Not super complex but no single nighthawk is gonna do it and the unifi management interface does the job. I'm not cloudy though.
Probably not big by US standards, but WiFi attenuation across multiple floors is such that an AP in the living room won't provide any decent signal one floor straight up. Depends on the materials and layout of your house...
This also means you can re-use a frequency with just one floor in between and no issues, and with a horizontally directional antenna, possibly even on adjacent floors.
I run two AP's hard wired to the PoE switch in my closet. These AP's being in the hallways on opposite sides of my home. I run them at lower power so I don't have an excessive amount of RF blasting into neighbor's homes, but I still get good signal quality to/from each AP. Because I now have two AP's running on different channels I've effectively doubled my network throughput overall.
One important thing to think about when planning your WiFi deployment is if you have things that have poor connectivity, everything on that channel suffers. I can have several devices running at several hundred megabits of quality, but a single device being really slow bogs down the channel and suddenly everything else starts getting lots of jitter and overall poor network performance despite most devices having good signal quality. Also, your device may show it has good signal strength but it might be poor quality (bad SNR) so in reality its a poor link speed. Having things physically closer usually results in better average SNR, meaning higher speeds for everything on the channel.
Also, as others have mentioned 5GHz might make it through a wall without a lot of stuff in it, but its not going to penetrate very well through several walls. Having my AP's in the hallways means there's usually only one wall with minimal stuff in it between a device and the AP, so each device usually reports at least several hundred megabits of throughput possible.
I feel the same way - my Nighthawk is going strong with custom firmware, but my friends with Ubiquiti gear try to get me to replace it with a bunch of Unifi stuff every time I talk to them.
Depends a lot on the house. My house is <2000 sqft, but signal, especially 5Ghz propagates poorly though old school plaster walls.
It wasn’t a problem until covid when multiple meeting or other streams just performed poorly on a marginal network. The Ubiquiti gear made it easier to run antennas for optimal signal.
The hot thing to do is to shit on them, but I’ll be sticking with it. They’ll emerge better from this crisis and if you think that any competitor in this price point is better, you’re delusional.
COVID had me setting up more UniFi APs. It held up incredibly well for moving large files across VPNs and running multiple Zooms for work places and school.
COVID must have been a massive boost to their bottom line.
I’m no market analyst, but the last year, even including the last week, has been very good to Ubiquiti.
My house had a problem since the cable came in on one corner of my house, and my office was on the other side. Browsing was ok but things like video calls suffered, at least until I went with a Unifi BeaconHD.
Getting signal to devices isn’t a problem, but it’s not easy having an AP receive signal from a low power device. Multiple APs is the way to go in my experience.
People want a power-user Meraki for the home that isn't tied to a cloud service. It's really as simple as that. Ubiquiti gave them that until they didn't. And now the inevitable breach has occurred and users are looking for a replacement.
Its pretty simple, having each device individually managed is archaic, a pain in the ass and there is no technical reason why it has to be that way.
Skipping wifi 6 seems like a smart move, with 6E on the horizon. It includes all the things that should have been part of the standard in the first place, so why get your hardware certified for 6, if you have to get it recertified for 6E anyway shortly after?
6 doesn't add very much over 5 in real world setups, very few devices even support 802.11ax yet, and the bleeding edge has never been Mikrotik's target segment.
6E gear is not really available anywhere yet, so it's really only an issue for people who just have to have the latest gear at all times. For the majority of people, 802.11ac/wifi 5 is what their hardware supports, so that's what they need.
As far as I know, that concerns 802.11k/v/r, MU-MIMO and beam forming, which many other 802.11ac devices also don't support, so it doesn't bother me. Then again, I'm not running an enterprise setup and I've never been one to meticulously make sure I get every single feature in the world on a spec sheet.
The hAP AC² serves my home networking needs quite well, with an additional AP to better cover the whole apartment.
It's an interesting idea to have a single pane of glass management experience for OpenWRT - given that all config is under UCI [0] it seems very possible. One of the things on my todo list is to try and get Nix to push config to my Unifi APs when I flash them with OpenWRT.
I know TP-Link is no Ubiquiti, but I run two identical small networks (VR-2100 routers with RE-200v4 extenders running in mesh mode), and it's pretty solid so far.
You can access your network from Tether app via cloud if you wish, too. When you enable Mesh, everything is controlled via the router. You don't need to manage anything on the extenders.
RE200 can work as an AP if you can get them a CAT5, or can provide wireless to Ethernet capability. I don't need home-wide VLANs and other exotic stuff (for a home network), but you can adjust QoS on the router in three levels and it has an embedded OpenVPN server if you fancy.
While not network related, you can temporarily or permanently turn off all LEDs on the devices so they don't create any light pollution, something I love to have.
All in all it's a great package, for my home network, at least.
Keep an eye on the Cisco Small Business line - no subscription, firmware updates without an account (yes, I am still talking about Cisco) and while the management console is a bit weak, I'd wager Cisco will mature faster than UBNT can get their crap together at this point :p
> Google WiFi creating loops in the network when users try to do wired backhaul
That's very surprising to hear. The decades-old spanning tree protocol can prevent that. I in fact have a friend who has done the exact same thing (Google Wifi with wired backhaul) with no problems. It switches from 802.11s to STP with no problems.
During this week I've been playing around with replacing my USG with my existing home server - it already has two NICs - my first thought was to run OPNSense in a VM but nftables on NixOS seems to work well enough - there are a few examples floating online [0,1]. OpenBSD even supports the USG [2] but I couldn't think of much reason to keep the extra hardware.
The next thing I want to do is reflash my Unifi APs with OpenWRT [3] - the hardware is fine, but at that point I'll get all the support without the controller software.
My home environment is fairly basic so moving away isn't too hard - this would obviously be much harder for a small business...
That’s odd, the link works for me but the wiki was very slow earlier. From what I’ve read Ubiquiti have made it harder to flash new hardware, but even the new ax APs are supported by OpenWRT. There is a commit with some info - it seems there is a way to disable signature verification [0].
I _do_ run opnsense in a VM and am very happy with the setup. My requirements for APs are simple but hard to satisfy. Ceiling mount, PoE, present-day-best 802.11 standard, and openwrt-capable.
I had assumed a setup which had several VMs, with one being a PFSense or similar to be less secure than a standalone firewall. Reading about the pros and cons leads me to conclude that security in a virtual setup is just fine.
I mean, don't get me wrong, there absolutely _is_ somebody who's responsible for it, but I wouldn't place any money on Ubiquiti being able to figure out who it really was.
They want to brush this under the rug as fast as they can, and that means using the opportunity to pin it on somebody that's been "problematic".
Given they were stupid enough to spin up some VMs, I doubt it was someone that knew what they had access to. A skilled attacker would stay dormant sucking up all data accessible via the AWS API (including s3 stuff) and potentially keep access to the infrastructure for years.
This kind of analysis is basically worthless because you don’t know whether they are operating at multiple levels of deception by, e.g., making you think they are a stupid script kiddie and that you successfully wiped them out.
If they had root access to an AWS account, this is exactly what you would expect.
If there's a cyber security firm that's been hired to provide analysis they're going to be combing through egress traffic to find anything suspicious. But, egress traffic is difficult and expensive to analyse.
Worse yet, the attackers could easily just sit there and not use their attack methods for a little while and start up their compromises in weeks or months. You couldn't be certain nothing's still there till you ripped the AWS resources out and replaced them.
And if it is happening, we might hear about that in a few years' time, if it's discovered, and if it's brought to light in circumstances that are conducive to the vendor making a public disclosure (eg. which are impossible to cover up).
Heh... no. I quit two years ago, well before all this happened. I have ideas about who this "Adam" is, and I also have some suspicions about who they're accusing as the culprit. But that's all they are. Hunches.
That may have worn thin, nowadays. The average response here would have been described as cynical in the past. The Russia/China scapegoat had been way overused to the point where I'm cynical every time it comes up probably even where it's actually true, one time in a hundred or whatever.
Nobody blames the NSA in these circumstances, ever.
> I'm just a guy who worked at Ubiquiti for a year
Would you be able to point to unofficial compatible operating systems for Ubiquiti devices? I want to remove Ubiquiti software from the devices I bought and paid for.
When I'm bored, I sometimes intentionally take comments out of context, just to see where they go, I know this isn't what you ment, but I like to pretend:
>Form your own opinion, I'm just a guy who worked at Ubiquiti for a year, raising all kinds of hell about the security, architectural, and operational problems that I saw while I was there.
You are a lawn man/woman.
Security problems: I have to show my badge EACH TIME I go to the bathroom
Architectural problems: these bricks are the WRONG COLOR!
Operational problems: The painters used the WRONG COLOR OF OFF WHITE!
Again, I know this isn't what you ment, but I enjoyed transposing a well written critique of their software from (presumably) a knowledgeable software guy into a lawn person in a jumpsuit.
I worked at Ubiquiti while you were there. I can confirm that the company was going downhill fast.
The US offices were starting to feel empty because so many people were leaving the company. Only place I've ever worked where engineers would quit before they got another job.
Saddest part was all the wasted potential. There were good engineers making good products at Ubiquiti only a few years ago. Once UniFi exploded in popularity the CEO started trying to micromanage everything and it all started falling apart.
Greed. 100% greed. While I was there, the CEO loved to just fly between offices (randomly) on his private jet. You never knew where he'd pop up, and that put everybody on edge, because when he was unhappy he tended to fire people in large chunks (and shut down entire offices). Every decision was motivated by how it affected the stock price.
I'm just an outsider looking in based on a short paragraph, but that doesn't strike me as greed. How does firing entire batches of people help the stock price? Anyone with more business acumen than a cat will understand that it doesn't. "Oh, that office made a mistake? Let's fire the lot of them so they'll learn how to do better next time!"
Based on this, it seems more like an asshole with some attitude problems rather than greed per se.
It’s very easy to say “greed” because we want to believe bad things are always the fault of someone’s personal moral failings. Hopefully the tech community will start to realize that when the same problems keep occurring for the same reasons, it points to a systemic failure.
My apologies for the language, but throwing away the advantage and further potential of the USA, in the interest of personal wealth and quarterly profits, is even more disgusting.
The majority of America’s management culture is horribly broken.
On the plus(?) side this management culture sometimes allows for easy external disruption.
It's how you do a text replacement in VIM, I believe it's s for substitute, /../ for the regular expression, and g for global, to substitute multiple instances.
It's unfortunate what seems to have happened to Ubiquiti. The idea of decent network hardware with a good UI that can support the prosumer to small business segment of the market has a lot going for it.
In the early days, it seemed like Ubiquiti was going to nail it and was building up a strong, loyal following as a result. Then came all the reports of quality problems, promised features never delivered, phoning-home, ads in UIs, the not just security breaches but cover-ups...
How the brand hasn't become toxic already is a mystery to me, yet look at the stock price tracker. It's been trending up for years and it has well over doubled in the past six months alone. Apparently investors aren't too worried about any potential consequences of all these reported problems.
The early days at Ubiquiti were good. I worked with a lot of good engineers and we shipped good work. The decline is a recent problem.
> How the brand hasn't become toxic already is a mystery to me, yet look at the stock price tracker. It's been trending up for years and it has well over doubled in the past six months alone.
This is your answer. No incentive to change. All of the bad engineering decisions have been rewarded by increasing stock price and continued sales.
Most of the original engineers have quit by now. I lost track of how many UniFi engineering leads joined and then quit after it started falling apart. Before I quit, I heard rumors that the CEO was making two separate teams work on the Dream Machine project separately, competing against each other. That made more people quit. I think they were trying to reboot engineering in foreign countries when I left because it felt like we were forgotten in the US offices.
>This is your answer. No incentive to change. All of the bad engineering decisions have been rewarded by increasing stock price and continued sales.
It'll come around, it just takes waaaaaaaay longer than you'd think for a slump in engineering quality to be reflected in the market. Especially with hardware.
We have a few publicly traded clients that we've worked with for decades (and by "decades" I mean longer than I've been alive). It's cyclical that they want our engineering to build new products when they're doing bad in the market, and once our work is released and gets them some success they'll design transfer back inhouse as aggressively as possible (their engineers aren't all bad, it's just not an engineering culture there). By the time we're out, they're still riding the upswing. Their management's institutional memory either doesn't see the cycle and/or they don't care beyond the next few quarterly reports.
What I'm trying to say is I know hurts to see your baby languish but it catches up to them, eventually.
IMO, the CEO had a bit of a Steve Jobs hero-worship complex, but only all the bad parts. I can absolutely see him putting two teams on the same project, and "may the best product win".
The team that "lost" would get canned, obviously (I saw it happen to two separate offices while I was there).
> IMO, the CEO had a bit of a Steve Jobs hero-worship complex, but only all the bad parts.
Part of me wishes Steve Jobs had never been brought back to Apple and died in obscurity. He's such a bad example. People idolize him, but his good parts can't be imitated, his bad parts can, and a lot of people can't seem to tell the difference.
Intel tried this too, according to an ex-Intel employee here. It's a management strategy intended to get the best result by inspiring competition. The problems it invites are the obvious, but the tradeoff may be justified in some scenarios.
It's also the premise of David Mamet's famous play Glengarry Glen Ross.
Google certainly seems to do this when it comes to chat applications. Ironically though, they've actually (arguably) lost marketshare - they went from gtalk being pretty widely used (in the late 2000s, early 2010s, as Android took off), to having a confused and fragmented ecosystem (Allo, Duo, Hangouts, Chat, Messaging), and it seems none of those have the same market penetration as the original did.
Perhaps internal competition to that extent simply confuses customers?
They essentially destroyed all competition (AIM, YIM, ICQ, MSN etc), the open source solution that would standardize chat (XMPP) and themselves. Making people just go and use proprietary solution like WhatsUp.
There’s an infamous anecdote with Jobs doing this. Tharanos had the same “two teams” story.
A lot of CEOs who think they’re the next Steve Jobs, don’t understand their own tech, and presume the solution to their technical problems is a lack of “motivation”.
Creating a skilled skunk works team to handle a critical problem is a great idea. Making two? And putting them in conflict? It’s like throwing your a steak to your dogs to have them fight over dinner. Idiocy.
I can see why the idea is tempting, ie testing multiple strategies and survival of the fittest. But in reality there are extreme downsides. Teams will lie and fudge data to get ahead. People dont trust their coworkers.
I think this is where strong technical leadership is needed. At some point someone needs to make a decision on the technical direction and have the conviction to stick with it.
I imagine it comes from some flawed business belief in the survival of the fittest. I've never heard a tech person advocate for it, I only ever hear it from business types.
Of the things I've seen reportedly happening at Ubiquiti, that one makes more sense than some.
Businesses put projects out to tender all the time, and other businesses that can provide what is wanted invest sometimes very considerable resources into putting in a bid, knowing that if they don't make the winning bid then those resources will mostly likely be completely wasted. Evidently it is still worth operating a business on that basis because the benefits when you do win outweigh the costs of the failed bids, and those costs might include reducing morale in a team who worked on a failed bid.
If that is the case across industries as a whole then economically it might make sense for a business to operate on the same basis internally for their Next Big Thing. Run multiple independent teams at the start, give them all the same brief, then see which team comes up with the most promising starting point. I don't see much of an argument for continuing the internal competition beyond the concept to prototype stage, though, unless perhaps it turned out that more than one team could produce a product that was viable in its own right without competing for the same market.
What do you suggest for someone leaning on an EdgeRouter Lite (with EdgeOS v1.10.11, staying far away from v2.x) and a Unifi UAP-AC-PRO access point?
The router will probably reliably carry me until saturating 1Gbps becomes a daily occurrence and the access point will be retired when WiFi 6E comes around (assuming Ubiquiti's WiFi 6E access points aren't required to connect to the cloud.)
Also in answer to sibling comments - you don't need to connect the UI software to the cloud. I have an Edgerouter SFP-X and a few AP lites. I recently added an 8 port Unifi switch for more PoE ports.
Following is to the best of my knowledge! Any ex-Unifi folks or other pros are welcome to correct me:
- The Edgerouter absolutely does not talk to ui.com (except check-for-updates). There's no remote control ability etc etc.
- The Unifi range can be controlled from the cloud, but via your Unifi Cloud Key. You can run this software yourself, without buying extra hardware. When it is not running there is no comms to the cloud. Run the software, configure things, stop the software - I run it in docker on an rpi4.
I think the brand isn’t toxic because of the state of the competition.
Even with this hack, their stuff is still the best available for home use. Netgear or Linksys consumer routers are awful. The mesh devices are okay, but serve of a different market.
The other stuff people recommend is often 2-3x the Unifi price and 2-3x more complicated to setup and configure.
Any ex-employees want to start a company making this stuff that doesn’t suck?
The other stuff people recommend is often 2-3x the Unifi price and 2-3x more complicated to setup and configure.
I don't know about 2-3x the price, at least not here in the UK. We looked into this when fitting out a new office with the networking essentials a couple of years ago, and Ubiquiti wasn't particularly attractive on headline prices compared to the other typical brands that get mentioned in that space (MikroTik, DrayTek, etc.).
However, the ability for non-networking experts to set something up quickly that does the job and doesn't have glaring security problems is definitely a competitive advantage in that prosumer to small business market. None of those other brands has a great UI that I've seen and they all tend to assume that anyone who wants to set up a couple of extra APs for a small office WiFi and a standard firewall for the Internet connection will be a pro-level network expert.
I think it would help a lot of people if better products/companies started to compete seriously on that front, and I have to think that with the SME market to fight for there is room to compete with the established names. After all, that is largely how Ubiquiti themselves broke into the market, or at least that's the perception I had at the time.
Who is "we"? You're talking about brands aimed at enterprise customers. I have no idea how much penetration Ubiquiti has managed to make into that market, but certainly around these parts its products are better known in the tier below that. The kind of organisation that is considering Ubiquiti IME probably wants significantly more functionality and scalability than home or entry-level small office gear but isn't working at enterprise scale and doesn't want to pay for it either. That organisation is unlikely to be considering the kinds of brands you mentioned as alternatives, and I rarely see any of those brands mentioned in discussions about alternatives to Ubiquiti.
I kept thinking that all the laments about Ubiquiti and others are enterprise-level stuff and are sysadmins' headaches, so was thankful I don't need to worry about it. But more and more I wonder how I managed to choose an Asus 5 GHz router by reviews, bought it secondhand, and now have it chugging along for something like eight years with only some hiccups in summers from heat. With no ‘cloud’ shenanigans.
Also, there are DD-WRT, OpenWRT and such. How comes people don't use those instead of whatever broken software the manufacturer bestows on them?
This always cracked me up. From what I can tell, he was a mid level dev on the Alexa web api team. He knew AWS sure, but he did not have the cred at all to justify the position and responsibility he was given at Ubiquiti.