I personally dislike with a passion every language or grammar that depends on white-space identation, especially if the designers were extremely opinioned to the degree that you can only use spaces (or even, a specific number of spaces per identation level) and not tabs.
It's not as a big of a deal in Python because as others have mentioned, you usually don't end up writing large functions to begin with, and tab identation is supported. It still means that if you try to send a code snippet to someone over email or Slack, IM, etc it may not work because whitepsace may be trimmed etc. With YAML, it's way worse for reasons the author outlined.
Something may look good on paper (or in screenshots) but practical considerations need to factored in when designing a grammar, and there are myriads pretty-formatting utilities that could be used to that end if one cares for that sort of thing (see also: clang-format).
Search means a lot of things, but even if we limit to mean web-search, as most people understand it, there is a lot more to it than the actual technology that matches queries to documents.
IR is for all intents and purposes a solved problem -- in fact it was solved a long time ago, and I highly recommend the seminal book “Managing Gigabytes”. I also recommend https://github.com/phaistos-networks/Trinity/wiki/IR-Search-... this page(disclaimer: I am maintaining it) for some interesting/important links to IRC technologies, developments, etc. While some novel ideas come out from time to time, the fundamentals haven’t changed -- progress there is incremental and mostly specific to different encoding schemes or ways to execute queries faster by using JIT or more cache-aware datastructures, etc.
Managing and queries documents based on keywords and boolean operators is one thing, and Lucene/Solr, and Trinity (https://github.com/phaistos-networks/Trinity) among other technologies can be used to take care of those challenges. But that’s the easy part (assuming you can do this fast enough, because you almost always can’t afford long-running queries):
- User Interfaces: Not just how results are presented, but also how users can construct or input queries. What options can be come available for filtering matches?
- Ranking: Precision is key, and rather simple formulas (tf/idf, BM25, etc) generally don’t work well for many/most domains. Furthermore, ranking is almost always not just about relevancy. It factors in static context scores (e.g document “popularity”), personalisation biases(how likely is it for user to mean Soccer or American Football for [football]),and other signals, fused together somehow to determine the final ranking of matched documents.
- Scale: Getting everything right is one thing, getting everything right at massive scale is whole different game. What may work on small scale(algorithms, technologies, services) may not work at all when you scale out.
- Everything else not directly related to search but either important or fundamental to a good experience/business: from matching queries to ads, to analytics, to autosuggestions, to training ML models to power all that, etc.
Web search is not a zero sum game. Bing makes over 3nb / year and while it may not have a chance to catch up with Google anytime soon, that’s a great business right there. Ditto for DDG. There are also companies that offer a different or better experience and access to datasets google doesn’t yet.
So, all told, search may be solved only in terms of the basic IR technology that makes it all work, and arguably a lot better than it used to be in terms of user interfaces, ranking, etc, but it will take a lot longer until those other aspects of web search may be considered ‘solved’.
I prefer silence or isolation most of the times. When I can’t get that or I am in the mood for music, I usually listen to random video games music, or tavern music in games ( on YouTube ).
I used to stay up as long as I can. I appreciate darkness and quiet, and this was naturally when I could get that. Nowadays though I sleep at around 10am, and I wake up at 06am. It’s almost as quiet, and almost as dark and I get to read, and shower and maybe have breakfast before rest of the family is awake at around 07:30. This turned out to work better for me all things considered. I am also 42 now so this may be in part because I am not that young anymore and the shifted priorities, habits, and needs are better served by this new schedule.
Studying codebases is one of my rewarding hobbies. I ‘ve learned more from that practice( https://medium.com/@markpapadakis/interesting-codebases-159f... ), than from most technical books I ‘ve read (by the way, most technical books aren’t particularly great).
Looks like the owner didn’t do this gracefully and didn’t care much about the significance of the Art to its creators and others who appreciated it likely way more than he did. That’s not very nice. On the other hand, if it is indeed his property he should have the right to do what he wants with it. Unless there was a contract of some kind, not a lawyer, I don’t understand how the judge could have sided with the artists who themselves were on a private property and ended up making claims to said property. Aren’t judges supposed to be impartial and make decisions based on the letter of the law ? Though I suppose the law in the us may protect the artists in this case.
> if it is indeed his property he should have the right to do what he wants with it.
The general principle is that you can do what you want with your property, but you must pay for any damages you cause in doing so.
The important question is whether you did cause those damages (maybe it was someone else's fault, or maybe the damages were inevitable and would have happened regardless of your action or inaction, or maybe you acted in good faith like any other person would have) which gives judges plenty to think about.
A better way to frame this would be to consider if the artists were sculptors who were allowed to leave their sculptures for free on this property. Pieces of art that are not attached to the building make it more clear that destroying the art would leave the building owner liable for damages.
Similarly, the artists who painted murals should have had the opportunity to remove their property from the building, which seems to be what they are arguing.
A verbal contract still counts as a contract though, so if the property owner had a verbal agreement that the artwork would not be destroyed, and he violated that agreement, I think it makes sense that he is liable for the damages. Seems somewhat similar to destroying a painting a tenant had hung up, you have to give them the opportunity to remove it first.
FWIW, we used MyRocks (went from InnoDB to MyISAM, to MyRocks) for some mySQL tables holding a few dozen million rows a few months ago.
InnoDB wasn't working out for us (INSERTs and SELECTs were too slow). MyISAM fared a lot better; INSERTs were a lot faster(obviously?), and SELECTs some 50% faster. But it too wasn't as great as we hoped it'd be. It got to where it was too slow for production use (some operations would take over 4s, which was a deal breaker for us).
We then switched to MyRocks. It was great initially -- much smaller on-disk footprint, INSERTs were fast, SELECTs were fast enough. But two weeks later it also got to where it was too slow. Slower than InnoDB and MyISAM even; also, our mySQL server would often starve for memory, and restarting it was the only practical way to “fix” it.
We would DELETE from those tables every few hours because we were only interested in the last 2 weeks worth of rows, so the dataset size was constrained/bounded, so slow downs weren't a result of tables getting larger.
In the end, we just gave up on MySQL, wrote our own thing that stores and accesses data on-disk directly. Disk footprint is over 2 orders of magnitude smaller, and all operations take constant time(no more than 20ms at 99pc), whereas in the past we ‘d get around 3s at best, and 10 or maybe 20s on average(not even at 99pc).
This is not about us doing anything “better” or about rolling your own alternative to generic datastores. It’s about MyRocks performance’s initially being good, but deteriorating very quickly - and how it compares with InnoDB and MyISAM, at least how it did for us. Also, using an RDBMS wasn't likely the right choice for what we were doing anyway, but for various reasons that's what we used.
It was in beta at the time, and I am sure there are tunables we could change to maybe get a better performance, but we didn’t really bother with any of that.
"We would DELETE from those tables every few hours because we were only interested in the last 2 weeks worth of rows, so the dataset size was constrained/bounded, so slow downs weren't a result of tables getting larger."
Yeah, this workload is hard on any LSM store, unless compaction is carefully tuned so that the tombstones are cleaned up often enough. Most people would recommend using partitioning...
Yes, we 'd insert new rows every 30 minutes or so. Also, we did try partitioning one of the 3 tables (which made sense) into multiple tables -- it didn't improve the situation at all.
Again, there are probably tunables and practices specific to MyRocks that we just didn't consider. We just didn't want to pursue this any further.
The only change that I'd recommend considering, is we use partition by list, instead of range, and keep a weeks worth of daily future partitions. This makes it so you don't have to reorganize the table. On the other hand it means if you get future data it can fail and if your partition jobs have issues for a week you can fail to load data all together.
Thank you -- we didn't use table partitioning because we didn't have good results in the past, though we should probably have tried it, and maybe it'd have worked well. We just kind of gave up on mySQL use for that specific problem after failing (again, maybe its entirely because of our inability to use mySQL "properly") to get good results and moved on to something else.
You new here? Reinventing the wheel and blogging about it has glory. Where is the glory in writing some SQL that just works? Moreover it makes the code base very readable and maintainable, making your job redundant.
I suspected that folks would think this is about rolling your own thing as opposed to relying on existing solutions, and I thought I was clear what I said was not about that — I just thought it would be somewhat valuable to someone how MyRocks compared to InnoDB and MyISAM for our use case. That’s just one datapoint.
Doesn’t mean others will have a similar experience(in terms of performance and scalability) with us. Obviously. YMMV.
I think you need to just expect and brace for a certain amount of rage on HN if you say that you did something outside the norm. It’s unfortunate that people with literally 1 or 2 sentences of context can be so judgmental and completely miss the point.
Note: in my original comment I rather deliberately did not call them the wrong choices, just weird. Hence asking if they'd engaged a DBA. They look like the kinds of choices that someone would make if they were just trying random things.
You could build your stack out as a full LAMP setup, all on a single box. As things get a little busy you could spin up a second server, running all the software, adding master<->master replication between the databases and split the load between them. Then you could add a third, or even a fourth, creating a nice database replication loop between them (1->2->3->4->1) to ensure all db instances remain in sync. It would work. You could probably get a nice shiny blog post out of it about how easy it is to horizontally scale a web application. After all, you didn't even need to deal with all that hassle of splitting up writes and reads off to master and slave servers as appropriate. "We easily scaled our application horizontally, you could too!"
Eventually you're going to run in to all sorts of hell with trying to keep the databases in sync. As your write load increases, the fleet won't be able to keep up, and everything on it will start fighting for IO. Approaching horizontal replication like that is such a horrible idea.
Alternative take, sadly from personal experience: At one job, I got asked to help out a customer who just couldn't get the performance they needed out of their database. They wanted me to help migrate them to a larger server, which I did. Not long afterwards they came back asking for help tuning their database, because it was still slow as molasses. Turns out it was a site trying to be like a yellow pages. Their central bottleneck came from a single field in a single table that detailed the businesses. That field was "categories", a nice FULL TEXT field. Every category had a four letter short code. They wanted to have businesses exist in multiple categories, and so what they'd do was have an alphabetically sorted list of short codes, semicolon separated, e.g. "DOCT;DENT;MEDI" (for a business that offered Doctor and Dentist services). When you went to look at the medical table, the query would do something roughly of the form "SELECT * FROM businesses WHERE categories LIKE '%MEDI;%';". This would have been about 2009. There was no way to index off that field, every query would have to do a full table scan of that column to find out every relevant business. It worked, and when just a small handful of people were using the site at the same time, everything was A-OK as the server was really powerful and brute force was viable. Add any more users and the thing would fall to its knees. It wasn't that MySQL was the problem, it wasn't, it was that they were using it wrong.
Switching engines wouldn't have fixed the fundamental flaw in the schema. I even showed them how they could solve all of their problems with a relatively simple schema change, but they wouldn't change things. They did spend a lot of time ranting about how it was MySQL's problem and "How can Google manage it but MySQL can't", when it really, absolutely and truly wasn't MySQL's fault. From the company perspective, everything was mostly great. They were paying for database servers far bigger than they needed, and paying for my time to read their rants, and have my advice ignored. You can lead a horse to the water, but you can't make it drink, I guess?
Just because you can use a tool one way, doesn't mean it's the right way to use it.
From the perspective of your average jack-of-all-trades sysadmin, who has had to dabble in DBA work from time to time, here's what seemed really strange to me:
InnoDB -> MyISAM. That was a really, really strange choice. When you mention that at the outset as a change made, that's the kind of choice that rings alarm bells in my head and makes me think "They really don't know what they're doing".
MyISAM is an older technology than InnoDB and has a large number of drawbacks to it. A few key ones:
* MyISAM writes lock up the entire table, vs InnoDB row level locking. The whole table becomes read only until the write is finished. You can only have one write or update happening at a time, forcing you in to effective serialisation for all writes. That introduces a nasty scaling limitation.
* MyISAM doesn't support transactions. It doesn't barf when it sees transaction related instructions, it just ignores them.
* MyISAM's crash recovery is next to negligible. InnoDB has transaction logs around and the like that help it to recover from a crash gracefully without data loss, along with a host of better approaches to data storage on disk.
* Development on it virtually ceased a long time ago. Lots of effort around query optimisations for modern architecture, multi-processing etc. etc. have gone in to InnoDB etc. and not in to MyISAM.
The only advantage MyISAM had over InnoDB for a while was lack of FULL TEXT column support in InnoDB, but that was added in version 5.6, which went RC about 5 years ago or so.
I can't imagine a single person who knows anything about MySQL considering that change.
You also indicated that the engines start out fine, but performance dropped over time, and that you were deleting data older than a certain length of time. That makes me think a few things:
1) Your tables were getting badly fragmented. By constantly deleting data older than a certain age, you were forcing reads and writes to be all over the place. The impact is worse if you're not using partitioned tables. Which leads into..
2) Not using MySQL native table partitioning. This introduces some major advantages with queries, allowing more parallelisation of various actions underneath it. It also has the advantage of limiting the scope of any operation, particularly index updates (as I understand it, index updates on a partitioned table only end up modifying the index for a single partition, rather than having to modify an index for the entire table).
Re: myISAM qualities:
1. Writes are infrequent; every 30 minutes or so we insert/update rows. Requests rate is very flow. This is for a specific report -- and said report was infrequently requested. Only updated by a single producer/process. All that means that locking wasn't a concern for us.
2. We didn't need transactions - if the producer would fail while it was executing the REPLACE statements, we 'd start it over and it wouldn't be a problem (idempotency)
3. We didn't care for crash recovery either -- if aything would go wrong, we 'd rebuild those tables (we only cared for 2 weeks or so worth of rows, rebuilding them wouldn't take long).
I think you ignore that, despite MyISAM's deficiencies, it's really fast if you don't care for the aforementioned properties/warranties provided by more modern engines. And it was -- for our dataset, it was almost twice as fast as InnoDB.
We have been running mySQL in production since release 3.x; we moved to it from mSQL. It may not mean much, but we know it mySQL well, at least some of our folks do.
As I said in another reply, we didn't use native table partitioning because we didn't get the expected benefits in a different use-case/dataset, but we certainly should have considered it.
What did you use MyRocks for and what kind of data store you ended up writing on your own?
Over 2 orders of magnitude reduction in disk footprint and going from 20s to 20ms in query performance - those are incredible improvements. Would love to learn more about how you pulled that off.
It's not that incredible; for most use-cases there is often an optimal(or "better") way to implement a solution that's specifically designed and implemented to support it, as opposed to relying on systems and designs that are broadly applicable/useful (e.g RDBMS).
We just figured out exactly what we wanted, stored the data in chunks, compressed(we used 2 var-int encoding schemes, and snappy compression), indices(skip-lists) for each file and each chunk and for queries, we parellize access to as files required across multiple OS threads (scatter-gather). In fact, I am sure we could have gotten better performance if we wanted to spend more time on that problem. It wasn't novel or particularly interesting or hard anyway. Just something that needed to be done to help us solve a problem.
I think this is just too many people. No matter how you spin this it’s just too many. Do Google or Microsoft have half as many working on their browsers? I’d be very surprised if they do. It should also be interesting to look back and find out how many people Netscape employed at its peak. I would love to know the distribution of those 1.2k people ( engineers, marketing, whatever ). Of course, it’s likely that I am too naive or clueless.
There isn't public information on this, but in the past I have heard that Chrome had a lot more engineers than Firefox (though it's a bit difficult to make direct comparisons because of scope differences). I suspect that's still true. I also heard the same thing about Edge at one point, but I suspect that may not be true now.
Comparisons with the past are meaningless because browsers are far more complex (and performant, and secure, and versatile, etc) than they used to be.
And sorry, but your gut estimates are not likely to be that accurate if you don't work in this area.
> There isn't public information on this, but in the past I have heard that Chrome had a lot more engineers than Firefox (though it's a bit difficult to make direct comparisons because of scope differences). I suspect that's still true. I also heard the same thing about Edge at one point, but I suspect that may not be true now.
Scope differences make it incredibly hard.
Plenty of people on the Blink team work on things like Skia, which neither the WebKit nor EdgeHTML teams have equivalents of (because they just leverage OS-specific APIs), or on their HTTP/TLS stack (again, neither WebKit nor EdgeHTML teams have equivalents of), etc.
Gecko is somewhere in a middle-ground; they rely on third party libraries (including Skia—does that make that part of the Chrome team Blink/Gecko developers?) but not quite as much as WebKit and EdgeHTML do.
Core web stuff? The Blink team is larger, yes, but by how much is a hard question.
FWIW, I believe the Gecko team is nowadays larger than the EdgeHTML team, but they're certainly close in size. (And close to the size of the old Presto team, which admittedly had a scope closer to that of the Blink team.)
The Chrome team has a lot of its own infra due to policy that a lot of Chromium stuff be public (its bug tracker system now code.google.com is dead and its CI infra are all done by the Chrome infra team).
I imagine to some limited degree it's true for WebKit (though I suspect they don't have anyone whose sole job is infra) given Apple don't have any general public bugtracker or CI system, and totally untrue for Edge.
I never worked on a browser, though I did implement the HTML5 spec, twice, and I wrote a JS engine(compiler, runtime). I know that it was a lot easier because I could count on infromation and other advances that were available to me, and anyone else really, at the time, the kind of resources and information I probably wouildn't have had access to it("whats javascript?") 20+ years earlier.
Browsers have evolved - because standards did, and expections along with them, but in the past they too were quite complex(see also Netscape's play: browsers, email, usenet client etc, suit) and again, it was harder then that it is nowadays to build such systems.
Which part of the HTML5 spec, though? A parser? Or the whole thing, including DOM APIs and all the features like media elements and so on?
Even an HTML5 parser is harder than it looks to implement to the level needed for a browser. You have to build a proper DOM, you have to be secure against all kinds of fuzzing, you have to implement off-main-thread parsing and speculative readahead for performance, you have integrate it correctly the HTML5 event loop for document.write() etc, you have to support innerHTML and so on.
Likewise building a JS engine is impressive, but I do not believe a Web-compatible competitive-performance JS engine can be built by any single person.
I feel like there's a tendency to put browser developers on a pedestal. Browsers are large and complicated, and no, a single person cannot write a browser competitive with Chrome from scratch. On the other hand, a single person can probably write a browser competitive with Dillo, if they are sufficiently motivated. Browser developers are just as competent and incompetent as other professional native programmers. There are several HTML5 parsers out there. JS engines are also relatively abundant, especially if you will settle for an interpreter. You can build such things yourself, too, if you have the time and want to hone your skills.
There's a huge amount of fairly boring work to write a DOM implementation (if you want half the APIs the web expects to work with the right semantics, you can't use an existing library), and layout is incredibly hard to get right (especially in a web compatible way). I think it's totally doable for a single person to do HTML + CSS2.1, with no scripting support, especially if you're happy to leverage existing libraries.
The big difference with most existing browser developers is the majority know a lot of the web platform inside out—but that's mostly a matter of having worked on an implementation for a long time and having to worry about edge-cases and interactions no web developer would ever think of. There's certainly a few who have achieved a huge amount for individual developers—but that's not a trait unique to the browser sphere.
The parser. Once for properly parsing HTML content, for a search engine, and the other for a framework used for saving pages (For an Instapaper like service). So it wasn't a big deal, but in both cases, a DOM was constructed and operations were executed against the DOM.
Nothing about that or the JS engine is impressive really. That was my point, more or less. Of course, there's a difference between building something that works, and something that works exceptionally works etc etc(all the stuff on top of that), but all told, I still don't think building a browser justifies assembling such a huge org, even if there is no reliance on third-party technologies.
I was toying with this idea in the past, where cells would support, in addition to primitives(numbers, strings, whatever) and functions, a new type which would be identified by a reverse domain name notation path/key (e.g org.hr.employees.cnt), and optionally a comma separated (key-value) pairs as arguments.
Whenever a value for such a unique (key, arguments) cell would be required, the application would collect all such that need updates -- or, you could click a button to update them all --, create a message that would contain all paths and arguments, and auth info (each user would require to authenticate with some centralized system, so that the auth key would be used for that RPC, and that the service could check for authentication and authorization ), call out to some service and get values for each of those cells (value could depend on the authenticated user, or could even be denied depending on privileges ).
Furthermore, the application would cache each such cell value (auth key, resource path, arguments digest, value) and it would periodically update them, or do so on demand (so that you can use it say on a plane, and refresh them when you land). That’s the gist of it, but there’s more to it and there are some optimization opportunities I am ignoring here, but it could work.
Then again, for all I know, there are already such services, or apps that kind of do this sort of thing.
I use the iPhone 6s. It was responsive, and battery would last at least half the day just fine. Right after the upgrade, batter would last for maybe 1 hour, tops, 30 minutes on average. No minor upgrade fixed the problem, and apparently its a widespread one too. I am getting either the 8+ or the X, but I still think this is ridiculous at best.
Half the day was normal to you?! What are you guys doing with your phones, I use some cheap chinesium android and it gives me easy 2 days of battery if not more.
I get substantially better battery life on my iPhones than my Androids, although I've been pretty happy with the Note 8 so far. Reasonably happy with 7+, curious to see what happens w/ the X.
Your battery is probably degraded, and no software update will fix it. Lion batteries just don‘t last forever.
You can get a replacement battery for your phone for €25. If you don’t want to do the replacement yourself, any 3rd party repair shop will replace it in 15 minutes. You can also go to an authorized service provider, but that will cost €100 and take much more time.
Half a day, while far from ideal, was long enough for my needs. However, as soon as I upgraded, and not a minute earlier, the battery problem manifested. Now, it could be a coincidence and it is indeed just a problem specific to the battery, but its at best suspicious - the timing is.
I will replace the batter anyway, and I am getting a new iPhone next week, I just don’t appreciate being “forced” to upgrade -- conveniently when the iPhone X came out, no less.
If your battery only lasted half a day before, it was already a severely degraded battery. The update will put a lot of strain on the battery (iPhone does a lot of re-indexing and other stuff after the upgrade), which might might have caused the battery to deteriorate even more.
And, as I said, you are not forced to upgrade. A battery replacement is cheap and quick and will solve all your problems and your phone will feel like new.
I also have the iPhone 6S, and I've also been having trouble with the battery. I've had to replace it twice. (First time Apple replaced the whole phone because of a manufacturing defect, second time I went to a 3rd party repair shop)
Sure, it would be nice if Apple products lasted forever without service. Unfortunately they don't. But servicing iPhones has become extremely cheap and extremely quick -- as long as you stay away from authorised service providers.
I am not concerned with the replacement costs -- it's just where I live, getting the battery replacement will likely mean that I have to ship it to another city(Athens), or, maybe, find some place in a different city in the island (Crete) who may be able to do it. Its a hassle and an inconvenience for me.
What is odd is that is seems to be related to rendering; sometimes when I scroll on either direction on Tweetbot or on Safari, the iPhone will stall for maybe 500ms and the battery will drop by 20% or so instantly. Other times, I will app-switch to Slack and it will do the same. Reading content in black background/white text on the other hand, doesn't seem to be draining the battery. So I think its safe to assume this is an upgrade related problem. Also, I have disabled spotlight indexing and pretty much everything else I could, and it's also been quite some time since the upgrade for any upgrade tasks to be running to completion still.
I had the stalls as well, and they went away after replacing the battery.
I’m pretty sure that there is no way to fix your phone with software. It’s a hardware problem. Just replace the battery, and it’ll work fine again.
I don’t know where you live in Crete, but typing “iphone repair cityname” in google should quickly get you to a place that can replace your battery in 15min.
>Right after the upgrade, batter would last for maybe 1 hour, tops, 30 minutes on average
What are you doing on your phone for that type of battery life? My phone (iPhone 6 Plus) has definitely been slower, but my battery life seems to be the same post upgrade. My phone is mostly idle throughout the day.
On iPhone 7 Plus and iPad Pro battery was awful for me until I did a totally clean install and manually added back apps. Massive bore but fixed battery problems.
There were definite performance and battery life issues with 11.0 most of which were fixed with 11.1 and improvements are coming as well with 11.2. For me personally it's on par with iOS 10 for battery life.
Same here with an iPhone 7. I am permanently in low power mode and even then I barely make it through the day - and I am hardly using the phone. It is ridiculous.
It's not as a big of a deal in Python because as others have mentioned, you usually don't end up writing large functions to begin with, and tab identation is supported. It still means that if you try to send a code snippet to someone over email or Slack, IM, etc it may not work because whitepsace may be trimmed etc. With YAML, it's way worse for reasons the author outlined.
Something may look good on paper (or in screenshots) but practical considerations need to factored in when designing a grammar, and there are myriads pretty-formatting utilities that could be used to that end if one cares for that sort of thing (see also: clang-format).