I have the utmost respect for Patrick and the team - and just read the same thing via mail.
But - as someone that _loved_ Microcorruption and followed StarFighter from day one, stalking the people on Twitter as to not miss the launch: That's not for me.
The game right now - as this article, this mail shows - requires you to learn about finance systems before you even start playing the game. It's a game where the rule book is - from this pov - too long and already uninteresting.
Good luck, I'll sit this one out and would love to see a future expansion in different directions. Or another Microcorruption, Stripe CTF etc.
I respect your opinion on this, but you might enjoy the tech tree which I keep describing as "low-level programming", which is what Erin and Thomas have been killing themselves over for the last several weeks. I haven't written about it much because, frankly, that's not my bag just like finance isn't your bag. (Granted, finance isn't my bag either, but I find myself warming to it after scratch-coding a stock exchange.)
Thomas describes that tech tree as "MicroCorruption the way we would have done it if we had dialed it to 11."
Interesting. I'm the opposite. I was sorta curious, so I started reading Flash Boys. It became immediately obvious that A: the author is fucking clueless, and B: I didn't know enough about how this system worked. The book called "Dark Pools" was a bit better (mainly as a light history of electronic trading). "Flash Boys: Not So Fast" was really fun, mainly because it systematically trashes the author of Flash Boys.
It was interesting enough that I've picked up a few books on market microstructure, that really get into the details. It's fascinating, not just technically, but from a standpoint of looking at humans. It's odd to see how such strange systems persisted for so long (like Patrick mentions in this post, the crazy 1/8th spreads that were gamed). Or in "Dark Pools", they discuss how some of these traders would call up Nasdaq wondering why a certain trade did or didn't work how it did. And they'd find out all sorts of strange rules. Which makes me wonder, wtf? Are Nasdaq's rules hidden? Or were people doing this as a living just too lazy to read the specs? [1] I can only guess how much easy money there was to be made 20 years ago.
I'm so looking forward to Stockfighter, enough that I started looking at FIX and OUCH (OUCH looks far faster since there's ~no silly parsing - just copy into a struct) just in case that's what they offered. It's too late for me to actually work in finance (far smarter people with more experience in math and compsci), but playing the game is still an option. (I've actually been thinking of how we might run some fake exchanges, federated, as a game.)
Anyways, I guess to each his own. But I'm guessing I'm not the only one that will find this scenario intriguing.
1: Even basic stuff, like market orders, are weird. Who the hell would ever submit an order that's literally saying "Sell at any price, including $0"? (Well I suppose now it's "current price minus circuit breaker limits".) Or "buy at any price up to $MAX"? Sure, most of the time it works out fine, but if you can't be bothered to come up with a limit, perhaps you shouldn't be buying or selling?
> Or were people doing this as a living just too lazy to read the specs?
Pretty much every domain I've worked in has stunned me with the number of people who do things for a living without reading the specs (often literally, or understanding the basics in areas where there aren't literal specs.) Both conscious cargo culting (where people know they are doing what they've learned to do from others without understanding the basics, and know what they don't know) and false knowledge (where people do things based on an understanding of the underlying process that is completely wrong, and so don't know what they don't know) are distressingly common.
And just to be clear, I don't hold myself to be superior in anyway. I avoid conscious cargo culting except when there is no practical choice; but I've certainly ran into places where I discovered I had been operating for a long time on false knowledge when good sources existed to correct that, but I didn't know that I had a gap that needed correcting.
When you've been treading water for so long just trying not to drown, it can be hard to focus on better swimming techniques when you get a moment of respite. Eventually, you can get into such a rhythm of almost drowning and recovering, that it can be hard to imagine anything else until you see it. Finally catching a glimpse of your neighbor who happens to just be merrily floating on his back can be revelatory.
He's one of the most honest and intelligent people I've ever had the pleasure to work with in my entire career. He was at Madison Tyler while I was there roughly 4 years until the Virtu merger (at which point many of us left).
This book, for what it's worth, is fantastic. If you're interested in nuts-and-bolts how-stuff-works, and less in the screenplay-friendly narratives that Michael Lewis wanted to peddle, it's a much better book than Flash Boys itself was.
"Kovac misfires by supposing the "front-running" condemned in "Flash Boys" refers to a particular kind of abuse, and not surprisingly it's the kind he plans to refute. In common usage there are at least two meanings to the term. One common meaning is as shorthand for "front-running demand," which is what happens when someone tries to figure out whether your mutual fund or your pension fund is buying or selling stock, realizes that it will change supply-and-demand dynamics for the stock, affecting the price, and front-runs that demand to profit.
Imagine if someone made a reasonable guess you needed milk, raced ahead of you to the only store in town to buy up all the milk before you got there, and then offered to sell you milk at a higher price than the store sold it to him; this is one example of front-running demand, just as described in "Flash Boys." (It's also called "trading ahead.") Another common meaning is "front-running a customer order," a type of insider trading, where a stock broker believes his own customer's order will change supply-and-demand and tries to profit from that; this is what Kovac apparently intends to refute, and it has nothing at all to do with "Flash Boys.""
This is a review written by someone who, by their own (repeated) admission, hasn't read past the introduction of the book; further, the premise of the review is that Kovac doesn't know what he's talking about vis a vis trading, which is something we know to be false.
Correct. Peter Kovac wrote the entire realtime risk and clearing system that is now live at Virtu (formerly California Applied Trading Science which split into EWT and Madison Tyler, which became Virtu. This is all public knowledge if you dig deep enough). Again, the source is I worked with him for several years.
"Front-running demand" is just anticipating where prices will go. That's what all competitive trading is. At its core, that's what business in general is. Observing supply and demand playing out through public information is completely natural.
The reason we have terms like "front-running" and "insider trading" and laws against those practices is because of ethical obligations between people. If I see a lot of women wearing blue dresses this season, and buy more anticipating they'll sell for a profit, that's just business. If my accountant also works for the shop on the next block and tells me they're about to buy lots of blue dresses this season, that's unethical.
Yep, I mention that in the first paragraph :). I loved the book. It was fantastic. Both for explaining stuff, as well as totally smacking down Lewis's nonsense.
It left me wanting to learn more. I'm currently reading Trading and Exchanges, as well as Market Microstructure in Practise (I wanted a newer book as stuff has changed in 2002 and 2010.)
How was Michael Lewis clueless? He gave a step by step description of how regulation in reaction to previous crises and the money gave rise to HFT and specifically that Banks get paid by HFTs to submit their orders to specific exchanges. HFT is a game of speed that arbitrages between dark pools and the various public exchanges that ultimately provides no value.
Among other things, there are times in the book where it is very unclear whether he understands execution of an order against the orderbook is atomic. For example, he posits that someone can, on a single exchange, hit a sentinel order placed by a market maker with a larger order and then have the market maker adjust the prices of orders behind the sentinel, in response to the execution of the sentinel order, prior to the original order hitting the orders behind the sentinel.
That can't happen. It might sound plausible, understandably, to someone who has never done concurrent programming and who doesn't understand that e.g. locks are a thing which exists in the world. I am attempting to restrain myself from being too critical here, because that's a higher level of technical knowledge than I would generally expect from a journalist writing for a general audience, but then again this journalist wrote a book-length treatise on how distributed systems and concurrent programming are screwing the American public without successfully demonstrating that he has a handle on either of these topics.
> he posits that someone can, on a single exchange, hit a sentinel order placed by a market maker with a larger order and then have the market maker adjust the prices of orders behind the sentinel, in response to the execution [..] prior to the original order hitting the orders behind the sentinel.
"That can't happen." True, technically. But what you described earlier is close what is called the "Last Look" ability.[0] It's highly unethical, because it provides market makers and large operators a privileged data feed and ability to effectively go back in time.
Im simplified terms: if an exchange provides LL, executions against standing orders from privileged parties would be flagged as "pending" for a short window and the privileged party is sent a message of the pending execution. During the time window, they are allowed to respond with a cancel, which will be performed in front of the execution. And if a market maker knows that an execution would result in a bad trade, it will then start issuing cancels for other standing orders too.
There's also a kernel of truth in the other part I quoted. I firmly believe that market makers can't adjust prices for their standing orders, but in some exchanges there are order types called "reductions", where the market maker is allowed to adjust the available quantity of a standing order. So instead of issuing a cancel + order, the market maker can simply drop the available amount in their standing order while still keeping the order itself in the book.
Last Look is a an issue on ForEx markets, but ForEx markets are already extremely dodgy. In many cases, your counterparty in the ForEx markets owns the exchange you are trading on. I've never encountered an equities or futures exchange that allows last look.
As for price reductions, some exchanges allow reductions, others model them as cancel + order but allow the new order to take up the priority of the old order (effectively a reduction) and still others model them as reductions, but you still lose priority (effectively a cancel and new order). That is largely immaterial though, because even if you allow reductions, if a large order hits a level it clears fully through that level before anyone can make any reductions. In fact, reductions aren't a problem, increases are because it would allow someone to sit in front of the priority queue until a level is fully cleared.
Something that did happen on some exchanges, which is closer to what you are getting at, is that on some exchanges the fill feed is less latent than the market feed. That is you would get notified that an order executed before that order was disseminated on the order book. This would allow you faster insight into the market, so people did use sentinel orders for that, but if a single order was hitting a level enough to clear it those sentinel orders wouldn't help you get out of the way on existing orders at that level, they would get filled as well. But it could be a leading indicator of multi-level sweeps. Largely that wasn't desired functionality on the exchanges side, just a by product of their architectures, but people whose job it is to understand how the exchange behaves, clearly had advantages there over those whose job is not that.
In this case, a sentinel order exploits the speed differences between different data feeds. So if an exchange sends data on executed trades out slowly, but notifies traders of their own trades in a fast way, then you place a small order so you can see market activity faster than the general activity feed.
The software engineer has a habit of imagining bow he would design a system, and then assuming the real system is built without any of the flaws he could design against.
His examples are just flat-out wonky. For instance, his key opener is a trader that is shocked to find out that when he buys tens of thousands of shares, the price, gasp, moves.
Or another one, where the price moves before the guy hits enter. So either he's saying HFT's have trojan'd traders' workstations or ...? Instead of investigating this, he just writes more "ooh spooky" lines.
At one point, he spins regulations requiring customer orders use the best price as a bad thing. Basically, you get the feeling from his book that he'd feel better if we went back to specialists on the floor with quotes in 1/8th, doing stuff you can understand as a human. That somehow, all this tech is bad. Hell he even makes fun of colocation - not adding multiple third-party providers of fibre to a connection to make it faster and less likely to break is obviously evil because... well he says it so ominously it must be true!
He doesn't say HFT provides no value. He says everything's a huge conspiracy (involving, apparently, everyone) and HFT is stealing money. He fails to realise there's risk. Instead he makes it out like HFTs can just buy so fast that money falls out (that if they buy at one price, the only possible outcome is a sale at a higher price, that the market can never move in the other direction). He even gives them magical capabilities, implying that they can determine total order sizes from a single trade.
That part about Brad Katsuyama in the beginning made me so angry. Here's a 'hero' trader making literally millions of dollars per year, and he has no idea how the stock market actually works.
It would be like UPS getting pissed off at Fedex for using this crazy contraption called the airplane to get freight overseas faster.
> where the price moves before the guy hits enter.
Can you give a page # reference for this? In the scene I recall, the price is stable on all exchanges for minutes--until they hit enter on very small order to BATS, at which point the prices increase on the exchanges further away.
“I hadn’t even hit Execute,” says the hedge fund president. “I hadn’t done anything but put in a ticker symbol and a quantity to buy. And the market popped.”
From location 142/505. And this is supposed to be the guy in charge of a $9B fund. The next page continues to state that all the big players (like Vanguard) have no clue how markets work. As if these evil HFT wizards live in another dimension where you can't just, I dunno, interview them. Or hire them. Or buy their companies.
The lulz never stop. Next paragraph has an amazing quote about colocation "What the fuck??!! That's got to be illegal!"
I guess if you act incredulous, insulted, upset, and bewildered, any mundane thing can turn into a sinister conspiracy. The sad part is that people are taking this book seriously, and politicians even seem to be "addressing" it as a campaigning point? Sigh.
The reason the price moved is because his trades were being front run. Otherwise the market had said there were 10k shares at X but when he tried to buy them he could not because an HFT bought those shares before he did.
No, no, no. The reason the price moved is because market makers saw the trades on one exchange, then updated their quotes on others. This is normal, regular, to-be-expected, good behaviours. Calling it front running just shows a lack of basic understanding. Or in Brad/Lewis case, probably an ulterior motive (selling Thor or promoting IEX).
In fact, the book even shows it's not front running because Thor just delays orders so they hit all market makers simultaneously, so they cannot react.
There's no "market", but a group of exchanges. The orders to a single exchange execute immediately; there's no way to do what you're saying. And across exchanges, this behavior is exactly like it should be. Here's how to verify: increase the times.
Suppose the trader moving a big block sent an order to one market to buy up 10K shares. Then he sits for an hour. In that scenario, do we expect the price to stay the same? Of course not! Everyone sees the one order on that one market and reacts (canceling their outstanding quotes on other markets, for instance). That's all that's going on, except faster than humans can react. Why is it suddenly bad when a computer does it?
Front running is well defined. Front running is not just acting like a normal intelligent person and updating your quotes (or even trading) after a large order executes.
Cliff Asness on how HFT has benefited retail investors.
How HFT has changed the allocation of the pie between various market professionals is hard to say. But there has been one unambiguous winner, the retail investors who trade for themselves. Their small orders are a perfect match for today's narrow bid-offer spread, small average-trade-size market. For the first time in history, Main Street might have it rigged against Wall Street.
The only reason retail orders actually get matched up well is because they're flagged as retail, are known to contain less information, and are a statistically good bet to transact against. Fundamentally, this was around before HFT and still is. The funny thing is that even if it's helping them in this small way, the whole idea of retail investing is some of the best marketing in the face of contradictory facts I've ever seen.
Please elaborate. If an order makes it to an exchange and that exchange has a bunch of small orders at all times, how is the retail-ness of the order relevant at all?
I understand that at other parts of the order chain this might matter. But inside the matching engine there isn't "retail priority".
Inside the matching engine, there's a sitting limit order on the limit order book flagged as a retail order, just off the bid/ask. Whereas you may not have taken the other side of that resting order before, you might now because you know it has extra informational value attached. Dealers will pay exchanges for the right to access this information, either outright or in reduced liquidity rebates they would normally otherwise receive (normally outright to my understanding).
"A U.S. Government Accountability Office study in 2005 showed that bid-ask spreads compressed by as much as 80% immediately following decimalization, well before high frequency trading came to dominate markets, and well before this author's time frame."
I have no personal experience in this, but I've heard that the order book is now very "shallow" due to HFT, such that it's hard to sell/buy a meaningful quantity of stocks at the market price. Does that make sense?
I don't buy that the order books are shallower than previously. I'd love to see evidence of it. People also complain that 'as soon as the markets move, the HFT firms get out of the way and the liquidity disappears.' As if the humans in the pits were making markets during big moves and getting run over (losing money) during high volatility. Give me a break.
The bid ask spread is a measure of how far behind you are the moment you buy a stock. If the ask is $10 but the bid is $9.75 you've lost 0.25 per share the second you make a trade. If the spread is only $0.01 (as is now common) you've only lost a penny. So smaller spreads are great.
It's true that markets now respond faster to large sales so if you're a hedge fund trying to unload $BIGNUM shares of MSFT you'll have a harder time. But that's good for everyone else. If that hedge fund has proprietary information that MSFT is undervalued you want that information getting out to the market as fast as possible leading to more efficient prices.
The bid ask spread is a measure of how far behind you are the moment you buy a stock. If the ask is $10 but the bid is $9.75 you've lost 0.25 per share the second you make a trade.
That makes some sense, but who have you lost it to?
I was thinking about it the other day, and it seems that the "liquidity is good" argument can make some sense if people are not willing to make a transaction immediately. If I want to sell a bag of apple for 1$ _right now_, and someone else is willing to buy that bag for up to 1.50$ in a week, then I can appreciate that there's a middleman (providing liquidity) willing to hold the bag for me during that time and that he makes some profit from the spread. However, if the buyer and seller are both willing to make the trade at roughly the same time, then it's much better if they do it themselves (splitting the 50 cents of value amongst themselves) than if a middleman gets into the picture.
You've lost it to a market making who is charing you for providing a service. Whenever you trade with a market maker you are buying liquidity (which used to be expensive but is now a lot cheaper). You're saying "I want to take the price you are offering right now and am not willing to take the risk that someone else will never come along to trade with me at a slightly better price."
If you don't want to trade with a market maker you can avoid purchasing their liquidity by placing a limit order that doesn't cross the spread. You'll then wait until someone else comes along to trade with you. Of course, you're taking a risk that the stock price will move away from you and you'll never trade. Normally this is the risk the market maker takes for you (which is why you pay him) but you can always do it yourself if you want.
In that comic the first caveman is getting a price that is right in front of him. He's not willing to wander around hoping to find a better price in another cave. That's a service that the 2nd caveman is performing for him. That's why the 2nd caveman gets paid.
The key thing to realize is that market makers don't just "get into the picture" by muscling their way in when other people don't want them there. Think of them off to the side selling liquidity for anyone who wants to walk up to them and buy it.
Thank you for the detailed explanation, I get the point about liquidity and traditional market makers, that makes sense.
But isn't the point about HFT (at least, the part that is described in the comic) being "insider information" reasonable? Some people are willing to sell now for a low price, and some people are willing to wait longer for a better price, but I don't think anyone is unwilling to wait 1 millisecond for a significantly better price, no? Does liquidity on the millisecond scale make sense?
> He's not willing to wander around hoping to find a better price in another cave. That's a service that the 2nd caveman is performing for him.
I can see that, and in some contexts it's a very valuable service.
But suppose I'm caveman 1. I know that CaveBob is a middleman. Everytime he comes to see me, it's because he knows something I don't, and I end up regretting my trade with him because a few hours later, I invariably realize that I could have made a much better trade. Then I'll stop trading with CaveBob, of course, and I'll naturally bump into those needing my meat a few hours later.
In a stock exchange, I can't stop trading with CaveBob's, as far as I know. If a stock market without market makers and HFT rises by 5% every year, then by participating, I can hope to make as much if I'm average at trading. In a market with HFT, they will be taking part of the 5% with very low risk (through technological advantage), so it's not clear what's left, and the decision to participate is not so clear.
> but I don't think anyone is unwilling to wait 1 millisecond for a significantly better price
Two things here:
1) Is 1 penny better when trading a stock that costs $50 bucks really significant?
2) The issues is that you aren't guaranteed to be able to trade at a better price 1 millisecond from now. Maybe someone comes along and takes your price or maybe the stock moves away from you and you can never ever get that price again ever (or even the price you could have gotten from the market maker). It's a risk. And it's a risk you are free to take, or it's a risk you can pay the market maker to take for you. Your choice.
> In a stock exchange, I can't stop trading with CaveBob's, as far as I know.
You can absolutely stop trading with CaveBob! You can do this, as I said, by "placing a limit order that doesn't cross the spread." So if Bob is selling $STOCK for $10, and buying for $9.95 and you want to buy you can place a limit order to buy for $9.95. CaveBob won't sell to you at that price but maybe someone else will come along later to take it from you.
But again, it's a risk.
It's also a risk with an adverse selection problem if you think about it.
> You can absolutely stop trading with CaveBob! You can do this, as I said, by "placing a limit order that doesn't cross the spread." So if Bob is selling $STOCK for $10, and buying for $9.95 and you want to buy you can place a limit order to buy for $9.95. CaveBob won't sell to you at that price but maybe someone else will come along later to take it from you.
Does that solve the issue? Suppose that I'm usually selling meat at 10$/kilo, but the buying side is at 9.50. Then when there's a catastrophe (some other vendor burned down), I could sell them at a higher price, 20$/kilo. If the news reach the HFT 1 millisecond before it reaches me, they can buy it at 10$ from me, then sell it at 20$/kilo for a big profit, but I would much rather have sold it at 20$/kilo myself. Is there any limit order that prevents that?
No. If you put in an order you're willing to sell at $10 (unless you notice the news about the catastrophe and cancel your order fast enough)[1], you're selling at $10. But if, instead, you just take the market price (as provided by the market maker) whenever you want to sell then as soon as the information enters the market you'll end up selling at $20 (or $19.95 or whatever).
This is one of the other services that you are paying the market maker for: price discovery. If you want you can do this job on your own. You can expend a lot of effort to keep up with all the latest news so you know what price you should be selling at. Or you can let market makers all compete to buy from you at the best price taking into account all of the latest information. In your meat example, keep in mind that there isn't just one trader who will buy from you for $10 to sell at $20. They're all competing to buy from you for the best possible price so they will very very quickly bid the price up to something very very close to $20.
This is why most people love market makers! If you trade with them you can be assured that you are (more or less) getting the best possible price taking into account all of the latest information.
Going back to your comic, this is what's happening in frame two. The ship arriving (for some unclear reason) is changing the price of neckerchiefs. It's generally to everyone's benefit if this new information enters the market as fast as possible. Rather than thinking something shady is going on we should think this is great, and feel good about paying them a few pennies to keep the information flowing quickly.
[1] Which is the exact same risk liquidity sellers normally take! This is why, in flash boys, Brad Katsuyama sees the price move when he tries to make a large trade. All the people selling liquidity are worried that there is some new piece of information they don't know about yet so they move their prices so they don't get run over.
This is why you sometimes see complaints that so many orders from HFT traders are cancelled. They're constantly monitoring all kinds of news sources so that they can keep their prices accurate. The fact that they're cancelling orders isn't a sign of shadiness, it's just a sign that prices are moving.
> Or you can let market makers all compete to buy from you at the best price taking into account all of the latest information. In your meat example, keep in mind that there isn't just one trader who will buy from you for $10 to sell at $20. They're all competing to buy from you for the best possible price so they will very very quickly bid the price up to something very very close to $20.
If I put a market order to sell meat at the market price of 10$ at 10AM (because I just produced a bunch more), then the first HFT to get the news of the catastrophe at 10AM + 1 microsecond will buy them from me at that price, but if I had gotten the news at the same time as they did, I would have withdrawn my market order and refused to trade at 10$.
Maybe HFTs/market makers serve many different purposes and we're talking past each other. I agree with the liquidity argument, to the extent that I understand it. Is getting the news slightly faster really providing value?
> You can expend a lot of effort to keep up with all the latest news.
The news that my competitors burned down doesn't require a lot of effort to keep up with in the age of the Internet, but it does require large amounts of capital to get it 1 microsecond before everybody else.
If trades were only allowed every 60 seconds like the comic suggests, to allow everyone time to process the news, what value would be lost? Should society rejoice when an HFT gets 10% closer to the speed of light?
> If I put a market order to sell meat at the market price of 10$ at 10AM
Yes, at 10AM (before news happens) you can take the $10 price. And, unfortunately for you, the world might change 1 microsecond later. And it sounds bad when you say "1 microsecond" but what if you say "5 minutes" or "1 hour" or "1 day"? You're probably going to feel just as bad about missing out on a better price even if the timescale is a human one instead of an electronic one.
There is no way to avoid this. The world can change after your sell your meat. It might change 1 microsecond later or it might change a week later. You chose to sell when you chose to sell. You can't say oopsies and complain if the world changes after you made your decision. The fact that the world can change after you make your trade has absolutely nothing to do with high speed trading.
Very high speed price discovery actually works in your favor here! At least the price jumps to $20 really really fast after the news now. Without computers it might take 5 or 10 minutes (or whatever). The sooner the news is incorporated the smaller the time window in which you can accidentally screw yourself.
While thinking about this, keep in mind also that the price might drop (maybe there is a bumper crop of cows). If you sold now for $10 but then the price dropped to $5 you'd feel great. When you sell now you're taking the risk of a price change (in either direction) off the table. You can't on one hand, avoid a price drop, but on the other hand think that you should get the benefits of a price rise.
> the first HFT to get the news of the catastrophe at 10AM + 1 microsecond will buy them from me at that price,
This is actually wrong and not what happens. The trade you're talking about happened at 10AM (before the news). You put in a market order which means "take whatever price is being offered right now." not "put my meat out for sale and wait for someone else to trade with me."
So no trade is happening at 10AM + 1 microsecond (after the news).
> If trades were only allowed every 60 seconds like the comic suggests
Ah yes. Everyone eventually hits on this idea, but it doesn't work. What happens if, at a given price, the # of buyers and sellers doesn't match up at the time you want to execute all the trades?
1. Execute the trades that put in their orders first. Well then you're back to where you were before.
2. Execute some % (less than 100%) of one side of the trades so that everything matches up. But then you create a very unstable situation. If I think that maybe only 50% of the trades are going to execute you've incentivized me to say I want to trade 2x of what I really want to trade. But wait, everyone else is also thinking the same thing. So maybe 4x? Repeat ad nauseam. This is a very unstable game-theory heavy sort of situation that's actually pretty dangerous.
And really even if you do this you haven't removed the need for high speed computers & communication. Lets say that a block closes at precisely noon. I definitely don't want to input my trade at 11:59:01. What if there is news in the next 59 seconds? I really want to wait until the last possible microsecond to input my trades for the block right? So...we're back to where we were before anyways.
The real world is continuous. It's not really possible to make markets based on the real world operate discretely.
This is actually wrong and not what happens. The trade you're talking about happened at 10AM (before the news). You put in a market order which means "take whatever price is being offered right now." not "put my meat out for sale and wait for someone else to trade with me."
That's totally right, I blundered my hypothetical scenario. Let me try again.
1. 10AM catastrophe happens
2. 10AM + 1 ms: news reach some HFT
3. 10AM + 2 ms: I put out a market order, ignorant of the news
4. 10:01AM: the news reach me
That's what I meant. But I suppose that given the competition between HFTs, the market price should have reached 20$ by then, so I get to sell it at the better price.
Does that mean that HFTs do not generally make money from getting the information before everybody else unless some non-HFT trader used a limit order?
If some HFT holds a lot of some company Y and gets the news early that Y had a bad quarter, won't they make money from that by selling Y to a non-HFT investor who hasn't received the press release yet? But maybe if no one but HFTs use limit order and everyone has a decent-speed internet connection, this is a vanishingly small occurrence?
Ah yes. Everyone eventually hits on this idea, but it doesn't work. What happens if, at a given price, the # of buyers and sellers doesn't match up at the time you want to execute all the trades?
1. Execute the trades that put in their orders first. Well then you're back to where you were before.
2. Execute some % (less than 100%) of one side of the trades so that everything matches up. But then you create a very unstable situation. If I think that maybe only 50% of the trades are going to execute you've incentivized me to say I want to trade 2x of what I really want to trade. But wait, everyone else is also thinking the same thing. So maybe 4x? Repeat ad nauseam. This is a very unstable game-theory heavy sort of situation that's actually pretty dangerous.
And really even if you do this you haven't removed the need for high speed computers & communication. Lets say that a block closes at precisely noon. I definitely don't want to input my trade at 11:59:01. What if there is news in the next 59 seconds? I really want to wait until the last possible microsecond to input my trades for the block right? So...we're back to where we were before anyways.
All good points. Although there's probably a game-theoretic solution to that...
> Does that mean that HFTs do not generally make money from getting the information before everybody
In general this is somewhat true. Market makers would actually love it if prices were very stable. Just sit there and buy at 9.99 and sell at 10.00 and make a penny a trade all day long. They need to respond very very quickly not so much to make money, but to avoid losing money when prices change.
And you haven't really "lost" anything since you never had the ability to trade at the theoretical future fair price to begin with--nobody does. As you mention, your best alternative is to work non-marketable limit orders. This, too, has a real price: adverse selection, risk of missing your fill, connectivity costs, time spent monitoring multiple markets and updating your order. If you're a large investment bank or fund, your cost to do this may be lower than crossing the spread. For the average investor, no way.
Indeed, I agree that "lost" isn't the best word choice here. Instead "spent" would be better. You've spent money paying for a service provided by the market maker. And due to the wonders of automation, the cost of that service has been drastically reduced.
You cannot really separate the two. If you support electronic trading, no matter what artificial restrictions you put on speed, the first person to get his order in will win, they will always compete on latency.
If you allowed arbitrary precision (or even 8 decimals) then there'd be a bit less emphasis on latency, right? As is, a penny is worth quite a bit so everyone has to queue up at the same price making speed win.
No there's a minimum spread that will exist because market makers won't make markets so tight that they are losing money. If they can't make money on the spread, then what's the point of posting that spread?
These discussions always happen whenever HFT is brought up. I've thought a LOT about how to remove a latency advantage from this game, and I just can't find a way to do it. I'd love to be proven wrong though. (And actually, so would the HFT shops - they hate the speed race).
The minimum spread isn't a fixed quantity, different traders will have different minimums. Right now there is a price floor, below which the price cannot fall, which is bad for all those who want to transact in the stock market.
As for the speed race, why not limit transactions so that they can only be executed on the dot, once every $TIME_STEP? You won't be able to see what orders others have been put out, so you don't gain that much of an advantage by sitting on a very low latency connection.
This is correct. In ultra-liquid stocks, with a finer tick increment, natural buyers and sellers could undercut the market-makers' spreads since they aren't concerned about making short-term profits. Market-makers could also compete with more diverse pricing.
It is not necessarily a bad thing to have a price floor. Removing the floor is a good thing for the parties that meet at the tighter spread, but there is an externality where somebody who wants the shares nearly as much has their order go unexecuted despite providing price discovery to the market by showing their order. Look at GOOGL for example. It is $700 with a $0.01 price increment. Somebody can jump in front of your order by paying just 0.0014% more, there is little incentive to show limit orders on the market, which this makes the order book very thin. There is probably a better balance where the tick is large enough to avoid people jumping your order, but small enough that traders can compete: https://www.sec.gov/comments/4-657/4657-40.pdf has more comments on these effects.
You can trade in a single price auction in almost every market worldwide. A large amount of volume in US equities trades in this way, so people already have this option, but many choose not to use it.
> As for the speed race, why not limit transactions so that they can only be executed on the dot, once every $TIME_STEP? You won't be able to see what orders others have been put out, so you don't gain that much of an advantage by sitting on a very low latency connection.
Sure you do, you're still first in whatever line there is; latency still matters no matter how much you try and slow execution speed down. Latency isn't about the speed of trading, it's about being first in line regardless (speed to the sales counter) of what speed the trades are limited to.
I should have been more clear here: all orders that come in in the time interval are resolved at the same time (or if that isn't possible, the order is randomized). It now doesn't matter if your order came in a minute or a second before the deadline.
Even if you randomize the order, you just open yourself up to gaming the system in other ways. Larger players with more resources can split large orders into many small orders and they be assured a certain percentage will happen earlier, and certain percentage will happen later, but if you have reason to believe that there isn't enough supply at the desired cost, you can use this to your advantage.
Randomization usually doesn't work in solving such issues. But making trades execute simultaneously doesn't need to randomize. You can just give everyone a pro rated amount. A firm that submits large order gets more shares filled. This isn't really game able is it? If a firm quotes a price for 10x the shares, the run the risk of actually trading all the shares at that price!
(There's also more complicated rules possibly, like trades only execute when there's a certain demand, or other liquidity based measures. Not sure they're good but it's an option.)
I'm not sure how well a pro-rated amount would work. Just because I'm willing to pay X for Y quantity of Z, doesn't mean my order makes sense if the quantity is scaled up or down, so forcing this on all market participants might have some interesting effects. Do you have other markets where this is the case, I would be interested in looking at this in a slightly less theoretical context before I felt comfortable with how the behavior would settle (not that I'm all that knowledgeable about financial markets in general).
> (There's also more complicated rules possibly, like trades only execute when there's a certain demand, or other liquidity based measures. Not sure they're good but it's an option.)
Well, I'm not sure any set of rules isn't able to be gamed in some manner. Larger players will have an advantage as long as rules don't specifically penalize them, as they have all the capabilities of smaller players plus extra capabilities provided by economies of scale and dedicated resources. Even when you specifically penalize specific participants to reduce "unfair" advantages, I think what generally happens is that some participants move into the niche just outside those penalty definitions, to eke some of the prior advantage in the new grey area. This seems to be the norm in almost all regulation, IMO. Not that the regulation is always useless, but you can't game a system without rules, and regulation by it's nature provides more rules to look for specific ways to exploit.
What stops an exchange from quantizing to, say, a second or so? Use proportional matching, or something even more clever like POSIT does. I'm sure there's a reason that this simple idea doesn't work but I haven't figured it out yet.
As far as more decimals, it doesn't sound plausible that all the market makers are gonna come up with the same price to 8 decimals. So the time priority will give way a bit to price priority.
> What stops an exchange from quantizing to, say, a second or so?
That wouldn't be fair, an order book is a line, such a rule would allow people to cut line and get shares at better prices before you allowing them to drive the price up or down even though you were in line before them.
> As far as more decimals, it doesn't sound plausible that all the market makers are gonna come up with the same price to 8 decimals. So the time priority will give way a bit to price priority.
That's how it used to be, compete on price first, but a minimum decimal put in place by law, the sub penny rule, to try and dampen HFT, it's only made latency more important since HFT traders can no longer compete on price. And he's right, there's always a minimum price below which there is no profit, so whoever posts that price first wins. Orders compete on time and price because that what is fair. Latency is inherently part of fair trading, you can't remove it and keep the market fair.
> That wouldn't be fair, an order book is a line, such a rule would allow people to cut line and get shares at better prices before you allowing them to drive the price up or down even though you were in line before them.
Well, I think the problem is less that the book is a line and it's not fair (the definition of fair would change in this system, it would be fair that you had the same chance as everyone else in that time frame), but more that you open yourself up to different problems, such as the larger players being able to split their orders up in to many smaller orders, to increase they chance a large percentage of their orders will go through before any supply has dried up.
There's actually another industry that has to deal with these problems, and especially with pecieved fairness. Online event ticketing. Ticketmaster runs a queue. AXS runs a randomized pool. Both are gamed. Interestingly, I think some brokers actually do colocate close to TM servers, which makes the parallel fairly obvious.
That wouldn't eliminate the problem. Demand changes over time, and supply is fixed. There are quite a few ways to invest in tickets. One is to buy all the tickets on an event that is expected to sellout immediately. Another is to find an event that you expect to sellout at some time before the event that will allow enough time for demand to build, and you buy tickets before it sells out. In the end, ticket prices change over time, therefore there will be a market. Even if the primary seller (e.g. TicketMaster) raises prices, if you bought at a time when they were cheaper, you can resell at a profit.
Event ticketing is a market, and as such there will be people that want to invest (even if it's just fans who intended to go and then decided they couldn't, or the money they could get for the tickets is too much to pass up). Scalpers (brokers) provide a service, liquidity. Often they go too far, and use technology to get tickets in unethical ways (ignoring site policies, bypassing captchas, using software to tie up thousands of tickets they have no intention of buying, etc). But the fact that they continue to exist is because there is demand (and now TicketMaster is in on the game as well, their TM+ service allows you to resell tickets on their site along-side their own, unsold tickets).
This isn't true. I've been reading up on this, and there are markets that sorta do exactly this. "Call markets" instead of continous markets, I think? (Or a hybrid with "crossing" markets?) One example is POSIT where you submit orders and the matching rules are not a time line, but use other criteria (price, size of order). Even some continous markets give size priority. And to the extent you can "cut in line", that's the whole point of a quantized trading period. Instead of rushing to the best price, anyone that orders at the same price will get filled equally. This kills the need for speed. (The trick is probably in making cancels take place in the next quantized period.)
As far as sub penny, presumably different firms would have different costs, so the level of profitability would be different. While dominated by exchange fees (or rebates), at 8 decimals there's still room to compete. (Perhaps with new order types that allow displaying your quote rounded, so as to not trigger another feedback loop where firms keep lowering by 0.00000001.)
You can only use other criteria, when the others differ. For the same size and same price order, what possible argument can be made to fill the last arriving order first and still claim a fair market? At some point, latency always matters. If you randomize so latency doesn't matter, you just create a system where those with more capital get more orders. However you want to "fix" it, you're still going to be making a system that isn't fair to someone. Most people would agree that waiting in line is the most fair option IMHO, and that's proven out by normal human behavior in the day to day world.
What possible argument? Simple: This exchange works on a 5-second quanta. All orders arriving within a quanta are treated equally. How is that unfair at all?
Like I said, there are systems that do exactly this.
> This exchange works on a 5-second quanta. All orders arriving within a quanta are treated equally. How is that unfair at all?
It's unfair to the trader who put his order in first just as it'd be unfair to walk up to a sales desk and cut in front of people who there before you. All this does is slow down trading, it doesn't make it any more fair and now you can play the latency game for putting your orders in at the last second while allowing more time to analyse the data. 5 seconds is a long time to a computer.
That such systems exist is beside the point, they are less fair than simply allowing the fastest trader to trade first.
You have a weird notion of fairness. Why reward investment in low latency above all other investment?
In any case, quantising to eg 1s, allowing arbitrary sub-penny bids, and fulfilling by prize and using time as a tiebreaker would be `most fair' in some sense:
The counterparty gets the best price. And people are only able to `cut in line' if they make a better price.
(I don't think the fairness of waiting in line is a fundamental thing. It's just one way for humans to allocate scarce resources, and it's not very economically efficient---it just works well with human psyche and social norms.)
> You have a weird notion of fairness. Why reward investment in low latency above all other investment?
I didn't make that claim, best price should be ahead of place in line. But place in line is the next most fair thing.
> In any case, quantising to eg 1s, allowing arbitrary sub-penny bids, and fulfilling by prize and using time as a tiebreaker would be `most fair' in some sense:
Remove quantising to 1s and I agree. Quantising makes it in no way more fair and removes the ability to use time as a tiebreaker. If time is the tie breaker, you don't need to quantisize as it adds nothing.
> (I don't think the fairness of waiting in line is a fundamental thing. It's just one way for humans to allocate scarce resources, and it's not very economically efficient---it just works well with human psyche and social norms.)
It works well with the human psyche and social norms precisely because it fits our innate ideas of fairness, and thus fairness is a fundamental thing. It's so fundamental in fact that it's evolved into us as experiments with monkeys and fairness show it to be innate.
It doesn't work though. Lets say there are 500 lots bid at a price and 700 lots offered at the same price. Which of those 700 lots on the offer gets filled once the 5-second call occurs?
Like any other pro-rata system? How is this difficult to make work? "Call markets" do exactly this. POSIT does this. Even NYSE does this in some cases to give priority to specialists.
The article is a bit suspicious. It talks about Thor, which timed orders to different exchanges so they'd arrive simultaneously. This is because a market maker would notice a large order on one exchange, and raise his price. This is totally and utterly expected - why should a market maker take undue risk? Increase the latency - suppose a user submitted a large buy order to Exchange1, waited an hour, then sent it to Exchange2. Is it not reasonable that the price would go up? So why is it suddenly so controversial when it happens on shorter timeframes? Why should RBC be able to make big trades and not have to deal with price impact? Why should market makers be left holding the bag?
Now whether or not HFT adds more value seems debatable. But it seems to be a fairly harmless result that naturally arises.
HFT has been around for a while longer than Reg NMS: http://www.bizjournals.com/kansascity/stories/2005/01/17/sma... describes an major player doing something that sounds similar to modern HFT in Jan 2005, presumably they'd been trading that way for a while.
> Even basic stuff, like market orders, are weird. Who the hell would ever submit an order that's literally saying "Sell at any price, including $0"? (Well I suppose now it's "current price minus circuit breaker limits".) Or "buy at any price up to $MAX"?
In MMORPG games, those kinds of orders on the auction houses are awesome because they mean someone wasn't paying attention. Profit for the person who finds them! :)
> ..in "Dark Pools", they discuss how some of these traders would call up Nasdaq wondering why a certain trade did or didn't work how it did. And they'd find out all sorts of strange rules. Which makes me wonder, wtf? Are Nasdaq's rules hidden? Or were people doing this as a living just too lazy to read the specs?
Sometimes the functionality is obscure or poorly documented, so it's difficult to understand how it will work, or you make an assumption that turns out to be wrong. Sometimes, different order types interact in unexpected ways (these are effectively complex systems).
Sometimes, the exchange just gets it wrong. On a number of occasions, I've investigated an incident and it turned out that there was a bug at the exchange's end.
Almost every fun project I've worked on required me to learn something new. Heck, the last level of the Stripe CTF3 required me to learn how to fake a distributed fault tolerant SQL database!
> The game right now - as this article, this mail shows - requires you to learn about finance systems before you even start playing the game. It's a game where the rule book is - from this pov - too long and already uninteresting.
Given that Starfighter's business model seems to be helping filter programming talent for recruiters, filtering out developers who aren't interested in programming financial systems is a good first step. My day job involves understanding the finance "rule book" and translating part of it into something that developers can understand and build to. But, even with my translation, the developers still need to be capable of understanding a certain amount about the "rule book" so they can delivery code/functionality that actually fulfils the requirements.
If a developer isn't interested in (or capable of) doing that, they're not going to be of interest to someone who's looking to recruit developers to work on finance systems.
That is not our business model. We are recruiters (or at least: we plan to figure out how to be). We do not provide information to outside recruiters or hiring managers.
Aren't you using the CTFs you run as a way of filtering out the best talent?
We intend to get in touch with very talented players. We’ll be interested in talking about the game, about what you’ve built, and about what you want to do in the future. If your future goals include “Get a better job!”, we can help make that happen. If not, no worries.
We have agreements with a small number of companies to find qualified engineers and introduce them to the companies. These clients pay us for introductions which result in placements.
He didn't dispute that, just that their model does not entail sharing your information. They aren't planning to be middle-men to recruiters, but to be a specialized form of recruiters themselves.
I'm with you on this one. When I first heard about this and signed up for the newsletter, I was genuinely excited. However, the more I hear about it, the more disinterested I am.
> These days the markets are decimalized — stocks trade in increments of a penny. (They are not allowed to trade in increments smaller than a penny, by federal regulation. This is unfortunate, because “the minimum spread is 0.01 dollars” is not any more rational than “the minimum spread is 0.125 dollars” — if someone is willing to provide liquidity for cheaper, we should encourage that.)
This is not true, at least in the US. HFT has made sub-penny trades a pretty common thing. Quotes are displayed in pennies, but using limit orders you frequently see sub penny trades for highly liquid tickers (AAPL for example).
> "A level is simply a price. Why not call it a price?
> I have a sneaking suspicion Wall Street invented many of
> these words to give customers the impression “This is
> all really, really difficult — pay us money and the
> complexity goes away.”)"
Its called a level because an order book has many different orders at different prices and quantities. If you funge all the quantites at the same price together you create a 'level', say 1100 shs of VODAFONE @ 107.5 . The next 'level' may then be 17 shares of VODAFONE
@ 108 followed by 5 shares @ 110
--
Another term for 'Level' is a trade term for what 'level' of market data you are paying for.
Level 1 Market Data corresponds to trade and price data... Price, bid, ask, trade price, trade quantity
Level 2 Market Data corresponds to when you want to know much more about what is happening on the exchange. You want to know how many orders are on each book and at what price/qty.
This is also commonly referred to as 'Depth of Book'
Dark Pools do not send out Level 2 data, while primrary exchanges do ( LSE, NYSE )
Any suggestions on what to read to get up on these details? I've started "Trading and Exchanges", "Algorithmic Trading and DMA", and "Market Microstructure in Practise". Does that sound like a good path?
You could be right (or wrong). But, if the point is to understand the market microstructure and dynamics (rather than endorse their philosophy), in my humble opinion, these articles do a good job there.
> Level 1 Market Data corresponds to trade and price data... Price, bid, ask, trade price, trade quantity
That's not correct. What you describe is simply a trade that appears on the Time & Sales (aka "the tape"). Level 1 represents the inside quote, meaning the highest bid and lowest offer. Level 2 represents the combined highest bids and lowest offers of all market participants (which obviously includes the L1 data).
The stock market is incredibly liquid: for any stock listed on an exchange, you can buy almost any quantity and sell almost any quantity, at any time the market is in session.
I think that sentence would make most professional equities traders chuckle. I mean, it's true-ish for some of the big etfs and some of the highest volume individual companies, but there are 8k+ listed symbols in the US and you can't just go buying and selling "almost any quantity" of most of them. Nobody really trades that way. Trading in size typically involves chopping a large order into smaller orders that get managed by a trader or really a trader's execution algorithm.
I'm not sure what exactly you mean by block (in terms of size), but let's say you have a stock and the average number of shares at the inside (best) price is X. If you are trying to trade 10X shares, you aren't just going to aggressively trade through multiple price levels all at once to get the trade done, and if you just place an order for 10X shares into the book you will find that the price starts moving away from your order price pretty quickly. For a high volume ETF like SPY, X can be thousands of shares, but for low volume mid/small caps it may be only a few hundred shares, which is not something I'd consider "incredibly liquid."
The sentence is close to literally true, but what I think you're questioning is the implied "[at a price close to the current price]" at the end of it.
As someone who wrote the whole tech stack of a hedge fund, this looks really interesting. The summary looks pretty good, though I've only skimmed it.
There's a lot of stuff in there for future games. Network stuff, low level memory stuff, backoffice, a million FIX engines to write against, optimal execution, distributed computing.
How long would a game take? And what other languages would you support?
It seems like it's utterly language agnostic. Patrick has said a few times that you could play entirely through curl if you really wanted to, or play manually through the trading interface. I'm planning on using this as a first "real[ish]" project for Haskell.
I'd much rather use something like Stockfighter, but built around experimentation; one that has no competition, and no recruiters looking over your shoulder, scrutinizing your work.
I tried to build one-- and quickly found myself far out of my league. I wish this article had been around back when I was trying to write it!
The non-competitive open-ended Stockfighter you're looking for is called Stockfighter. :)
We will have way, way, way too many people noodling around with this to scrutinize everyone's work. The linkage between the CTF and our recruiting work will be largely automated, and entirely opt-in: nobody that works for us is going to notice you until some automated thingy asks if you're on the market, and you tell the automated thingy "yeah, I'm interested in being recruited".
The overwhelming majority of people noodling with our CTFs, including most of the "best" players, will not be job candidates.
Fantastic read, and a great introduction to the subject matter. Kudos!
I'm also really enjoying the position taken on OSS here:
> I’d personally be very disappointed in an engineer who, in 2015, defaulted
> to scratchbuilding their own clients for every API they consumed.
> Starfighter loves OSS and the OSS culture. Go nuts.
I don't fully understand what's the main purpose of this "game" (edit: regarding recruiting)?
1. Is it just useful for getting noted by some companies and get invited to interviews? So basically it is equivalent of sending them CV and/or contacting relevant persons.
2. Is it supposed to replace the tech interviews (which suck) completely? In this case, yes, I can see the value of allocating significant time for these games.
I literally lack the ability to form coherent sentences about our business that don't somehow involve how to render a graph of AVR basic blocks in a React web app, is how little we're thinking about how the game interacts with recruiting right now.
We are going to get the CTF right, and then work from there to a sustainable recruiting business. We should have done it the other way around, but we didn't. :)
I adore that you're so up front about that, I'd love to one day be able to follow my passion quite so single mindedly, and leave worrying about how to make money from it until later.
Instead of "good at markets", I interpreted it as "good at understanding complex systems". I never assumed the companies they would contact would be entirely financially based, I think the idea is that if someone can figure this out the should essentially be vetted (after a good in-person meeting) as a good candidate for almost any project you want to throw at them.
To me, the stock-market aspect of this seems entirely because it's a large, complex, distributed system that's not entirely theoretical (i.e. has been real-world tested), can be modeled accurately enough in the game, and is complex enough to provide for some real challenging problems. There are likely quite a few candidates that could have been used instead of the stock market, but the stock market might be the ideal one in this case.
I think the recruiting thesis is that all manner of folks might meet that category, but a) they don't know it b) those looking for them can't find them.
The previous articles written by their team give a pretty good insight into it.
1. People just seem to love playing CTF-style programming games
2. People who are successful get put in pool A.
3. if Pool A people have checked the "I am looking for work" on their profile, then the starfighter team matches them up with people that want qualified candidates that will actually be able to do the work.
The thesis is that there are qualified people that want/need better work opportunities but traditional hiring processes filter out these people or don't even see them, and that Pool A of Candidates have PreQualified themselves just by playing the CTF.
It's like going on to World of Warcraft or EVE Online and hiring the top market traders / economic strategists for a real world position.
> but traditional hiring processes filter out these people
I was interested if this game replaces the "traditional hiring processes". But if the purpose of this game is to just make first contacts - I suppose the contacted pool of candidates are anyway put in that "traditional hiring processes".
From a prior discussion, the idea was supposed to be more of a "why vetted you quite well, and we have contacts, so now we can get you a lunch with a CTO" or some facsimile thereof (I doubt you'll talk to the CTO of a Google or Apple class company, but you might talk to someone with similar power to bypass some of the assessment and hiring policies like the CTO of a smaller company).
I agree 1000% that the best way to learn something is to start with a problem that interests you, and dive into it. The opposite way, where you start with a technology that sounds interesting and new, and learn it for the sake of knowing it never worked for me.
I wish I could trade with you; I love learning stuff but it's not so much about using it practically, as it is that I just want to understand it.
Unfortunately, for most stuff, at some point you simply must have a problem to solve to progress further, and that's where I get derailed. I have a hard time getting excited about the real-life problems, so I end up getting stuck in the beginning stages of new pursuits. I'll read textbooks, I'll memorize syntax, but I'm never drawn to getting my hands dirty.
What I like about this sort of games is that they provide targeted problems for people who aren't naturally drawn to this stuff in practice.
Just had the realization that Starfighter believes there's a huge market for the sort of talent matching they want to do in finance. Why else would they pick this topic?
We're actually not terribly interested in finance qua finance. We're interested in finance qua "it allows us to do anything in engineering in a fiction-consistent way, plugging into anything else."
Networks. Compiler theory. Language design. Web APIs. Caching layers. Core algorithms. Packet switch microcontrollers. FPGAs. CRUD apps. Big Data. Anti-abuse tooling. Application security. Machine learning. Any topic you possibly want to throw an engineer at, an investment bank is already throwing a team of engineers at.
Our Chapter 1 release includes levels which have you jailbreak multiple chips (AVR microcontrollers) on a custom handheld device given to minimally trusted gophers to own up a stock exchange through their REST API. And that isn't even a theoretical attack.
Additional factors which tipped the scales include "finance has a default goal function which is really easy to understand", "some subsets of financial work are surprisingly easy to model" (like a stock exchange is), and "finance has narrative choices which allow us to excite people without getting typecast as A Video Game For People Who Played Dungeons and Dragons In High School" (which is, for a variety of reasons, important to the team, including to me, the ex-DM).
Other strong contenders for our first game were "Like Dwarf Fortress but with code" (harder to build than a stock exchange; includes the word "dwarf") and "a nuclear power plant" (slight problem: we know nothing about nuclear power plants).
Any topic you possibly want to throw an engineer at, an investment bank is already throwing a team of engineers at.
This comment makes me feel that I must live in an alternate universe of software engineering. I've spent 99% of my career working on things that would be of no interest to an investment bank: GUIs, graphics, video, audio, GPU rendering, mobile platforms, CAD, content creation tools...
(It's not the first time I've had this feeling. At one point years ago, when I had been doing professional programming for maybe 3 years, a friend and colleague asked me a question about databases. I had to confess that I had never worked on software that interfaced with one; that all my work exclusively dealt with local files and network streams. He almost wouldn't believe it.)
I've spent 99% of my career working on things that would be of no interest to an investment bank:
One of the many interesting conversations I've had over the last few months is with a guy who leads a team of X00 developers out of one bank's 1X,000 developers, which does minimally GUIs, graphics, GPUs (for rendering and simulation), mobile platforms, and content creation tools. His #1 problem right now is that you are representative of many, many developers who don't know that his sort of group exists. I think he'd describe his job as "We're the thin edge of the wedge hitting the really hard problems so that our customers in the bank can focus development on the really valuable problems."
(I don't know if I'm cleared to talk about them until contract negotiation wraps. Sorry, not trying to be coy here. Take it on faith that many of the big guys run shops which have those needs and more besides.)
I'd like to provide a data point for this, btw. There are exchanges for sports gambling (we run one), and some of the most succesful betting syndicates run clusters of hundreds or even low thousands multi-GPU servers to crunch the incoming data on their real-time simulations. The results of this type of crunching are feeds, signaling and strategy shifts for their market makers.
I have no doubt investment banks, who operate on an order of magnitude more data, would have something similar. Even if majority of it was used for backtesting.
Huh, that's interesting. I certainly wouldn't have guessed that.
Now that I think about it, I have no idea what investment banks are really like as work environments. The caricature in my mind is a rigid hierarchy populated by career-minded sycophants who judge people based on the college they attended; bosses that would be played by Alec Baldwin in fiction; software stacks sold by Oracle salesmen on golf courses; senior architects who manage via UML drawings; workdays that extend to 9pm just because; and for a touch of fun, brokers who throw half-eaten lobsters at each other on Friday nights.
In all, the kind of place that would have absolutely no interest in hiring me and where I would never imagine working... Maybe my prejudices would need a reality check.
I think you're describing investment bankers circa 1995.
Tech in investment banks is full of, well, developers. I'm currently working on stuff that has X0,000 cores, X00 GPUs (compute though not rendering), web UI, large data aggregation and visualisation (around 1bn data points moving in near real time). I don't do fpga work but I sit near some chaps that do.
In fairness, I haven't done any CAD or audio at a bank. However, I did design sonar absorbing materials prior to working in finance and there's a surprising amount of similarity in the engineering I did then and the statistical sims we do now.
GUIs, graphics, video, GPU rendering: Finance professionals have a huge interest in visualizations and interfaces that let them understand and manipulate large quantities of data easily. Some of that data has (at least) near-real-time constraints.
Mobile platforms: ...and they want to do this all from wherever they are. :)
never underestimate the power of a gui and good graphics/visualisations to make people more productive and effective. even knowing next to nothing about finance, i can bet that they have teams working on the user interface that traders have to work with, as a competitive advantage.
I'm a little confused. It sounds like there are two games here: one is to win by algorithmic trading, and the other is to win by hacking the virtual devices/protocols/systems involved in the algorithmic trading. Is that the case?
I've been describing them as "tech trees" but am willing to hear any suggestions you have for making this more comprehensible.
Starfighter is divided into N tech trees, where as of the Chapter 1 release N = 2. Each has a distinct player-facing UI and, generally, some specialized backend systems, but they're all a part of the same product, behind the same login, running off the same message bus. Each tech tree has a series of levels: start at level 1, beat it and unlock level 2, etc. The levels do not necessarily progress linearly and trees can interact with each other in terms of level requirements, although I don't know if any levels requiring, etc, "passed $FOO in algorithmic trading and either $BAR or $BAZ from low-level programming" will ship in Chapter 1.
Each level has its own goals/victory conditions/etc. There may be interplay between levels, such that your past/simultaneous performance on one allows you to do X, Y, or Z advantageous thing in the other. Those interactions might happen across tech trees. I am saying "might" here because I haven't coded any level which actually does that but it came up at design meetings and our systems support it -- not sure if it will happen in Chapter 1 but I'd rate it as "highly likely" it happens in Chapter 2.
Chapters: we intend on doing a new content release (which adds to, rather than replaces, existing content) every 6~8 weeks, with a new tech tree introduced roughly every 2~3 chapters.
Would love anyone's thoughts on "How do I explain this succinctly such that I don't turn-off anyone who is less interested in a particular tech tree?"
Wow, you all are really biting off quite a hell of a project. You've just described some pretty complex game development, as well as the entire financial sub-game and the entire infrastructure for it all and with an eye towards helping programmers get better jobs.
I think I get what you mean by "tech tree", it's a phrase borrowed from real-time strategy games, yeah? Some of your other descriptions sound a little bit like what someone might find on a Steam game. If I hadn't played Sins of a Solar Empire (very occasionally, I don't have much time for it anymore), I think I'd be pretty lost.
Your current terminology is probably fine, it just needs some graphics to go along with it -- a basic chart or something. I bet you've already got such a thing in progress and it'll all make sense on launch.
> How do I explain this succinctly such that I don't turn-off anyone who is less interested in a particular tech tree?
Any chance you can partition your newsletter subscribers into "low level programmers" and "wannabe market hackers"? I think the misunderstanding is that you only talked about stock market concepts to an audience that included people that are only interested in low level programming.
Maybe it's just me, but it does seem like there's a better term waiting to be discovered for this than "tech tree". (I can't exactly tell, but it seems like you wouldn't mind a better term either).
Not a fan of "worlds" as proposed below. I have a feeling we're all just projecting our own favorite genre conventions here, but your different approaches remind me of different characters more than anything else. Ever play a game where you had a choice of approaches with different characters? Or had to switch between them and coordinate action or share information?
I'm sure there's plenty of people who possess the union of these skill sets (and of course, there will be plenty of actual human players that will), but you could describe your trees more as character growth/background/capabilities/backstory.
You would then have interesting story possibilities here—I would love the idea but I suspect you guys might consider story elements as potentially distracting.
I think "tech trees" is just the wrong word because it only will be familiar to people who've played Age of Empires. Maybe "worlds" instead, it's a more universal concept (I think).
Starfighter is a game with lots of worlds. In each world there's a whole bunch of levels. Finish one level, move to the next. There's no "last level" or "final boss" because every few weeks new levels appear. Sometimes whole new worlds are created too. Each world is totally different and getting good at one might not help you in another (think of it like a video game, where in some worlds you walk on land but in some you have to swim, and in some you have to walk upside down). If you finish all the worlds you might get offered a job so you can do more levels in real life.
"Tech tree" is a clear metaphor to anyone who has spent some time playing strategy games... but if you're looking to communicate the idea to non-gamers, then maybe "ladder" is a better term? It strongly suggests independence of each technology, which is accurate at least for your first release. The interdependence of a strategy-game tech tree is probably a closer match, but you risk losing some non-gaming readers. Bonus: your terminology of "levels" carries over nicely to a ladder-based system.
I run a mid-sized (5-10 employees) trading desk for an automated market-maker. My first thought upon reading about this game, not knowing what Starfighter is, was "holy shit, I'd love to hire someone who excelled at this or even attempted it."
I've worked with tons of extremely bright, well credentialed people who get lost in the weeds of analysis paralysis or trying to build the world's most complex model. Many fail. The best hires I've made internalize the nuts & bolts quickly, work hard to try many ideas, and are "smart enough." They're probably just as smart as the other guys actually but use intelligence as a tool rather than an end unto to itself.
I am definitely one of those people who would assume you have to be a genius to do work in finance because I imagine that if your product isn't perfect you will be eaten alive in the markets.
That said there is also one other thing that prevents me from looking at a career in that kind of business: I imagine the traders as the kind of people with no empathy who are just looking to metaphorically stab you in the back the second they get a dollar in bonus for doing so.
I've found that traders are no more or less empathetic than any other cross section of users you might encounter in other industries. There are some trading desks that are more bro'd up than others and each trading desk has it's own culture, but the same is true of startups, which correlate pretty well with small trading desks in lots of ways.
Further, the very concept of what a "trader" is has fundamentally altered in the last 15 years. A trader is just as likely to be a physicist or hold a PhD in signals processing today as they are to be your typical "Tall Guy With Aggression issues".
I'd also point out that lots of people co-mingle different industries. Trading is not investment banking, which is not asset management. There are interesting software problems and good people working in all those fields.
I should have phrased my point on smarts differently. You definitely have to be bright to make money trading; there is a baseline and it's pretty high. Beyond that, the most valuable things are attention to detail, working well under uncertainty where the best answer isn't obvious or even knowable, and simple hard work.
Aside from systematic errors like Knight, market-makers rarely blow up due to competition or a bad model, they just fade away. They make or lose money pretty slowly, they just do it consistently over time.
Aside from the most basic strategies there isn't one "right" way to price a market. It's an engineering problem like any other. How much risk do I accumulate? How should I manage this risk? How far apart should I set my bid and offer? Do I use a simplistic model with lower latency or a more complex one that's slower? Is it better to be more accurate and avoid bad trades or less accurate and build up priority in the order book by staying on a price longer?
A market-maker's objective function is to maximize total profit (turnover * average gain/$ traded) while controlling risk (volatility of P&L over time). Usually changing your answer to questions like the ones above will shift turnover one way and profit per trade & risk in the opposite direction. For example, if you quote your bid and offer very close together, you'll trade large market share and have less risk exposure since you get in and out of positions very quickly, but your profit per trade will be low (if traders arrive randomly, at best you will make the small spread between your bid and offer, in reality a fraction of that since they usually have a reason to trade against you). If you set your quotes wider, you'll make more per trade on large dislocations, but do less volume and hold more risk.
There are a lot of efficient points along this continuum and traders play to their strengths. Someone with little capital/risk tolerance, top-tier infrastructure, and forecasting models for the next price tick should play very differently than someone with a lot of capital, average infrastructure, and models that can forecast the next 10 minutes.
And you are right regarding bonuses, but all team efforts have this problem. Who creates value and deserves the best compensation in a SaSS startup? Is it the salespeople who get clients? The engineers who build the software? The Ops team that keeps it running reliably? I actually find trading to be a bit more meritocratic than other tech jobs: either you make money or you don't, and teams are smaller so it's easier to assign reward (or blame).
I'm sure that's one consideration, but there are a lot of other compelling reasons. It's fun, it's relatively accessible, it's easy to scale difficulty and it's easy to add competition. Performance is directly quantifiable along multiple axes and they can release multiplayer scenarios in the future.
I can't think of many options with all of these properties that also require the particular technical skills they're looking for. It feels like doing a market simulation reasonably well is actually easier than their other track (a security CTF).
One particularly nice aspect is how many skills an exchange simulator can test. You have low-level programming and optimization, interesting algorithmic problems and high-level data analysis. That's applicable to a lot of jobs outside of finance. As a game, it's surprisingly open-ended.
Personally, I'm glad they chose this. I actually wrote a market-making bot for a mock exchange as part of an internship. It was fun and a great learning experience and not just for finance. I'm glad somebody is bringing this experience to the public at large.
Patrick often makes the point that the majority of lucrative development work is in big, (usually) boring, business applications. That's where the money and employment is, so it makes sense to start there.
I don't think quantitative finance is what he means though. It's a small, selective sector which is well-known and at least a bit glamorous. There's certainly money there, but far more hopeful candidates than positions.
Markets are truly fascinating. However much you think you know, you don't know as much as you could. The whole idea of Stockfigher really resonated with my experience over the last year and a half. Markets are incredibly complex.
Around the time a large contract ended in Hong Kong, I was at a web development conference for new programmers getting out of a bootcamp-style program (8 weeks, program or die). They put me on a panel and asked me about what it was like to be a freelance web dev.
At one point, a guy raises his hand and starts asking about equity deals. He asks what it would take to get someone like me to work on a startup with him, no cash. Now, the market's rough for that sort of thing if you want experience. I told this guy that he would just "have to find some bozo who believed in the idea so much, he'd be willing to give up salary in exchange for massive risk. Who's going to do that?" We laughed a bit and moved on.
Well, I ate my hat: I'm that bozo. I was impressed with his knowledge, so I joined up with him and another expert he wrangled. The two of them had an impressive amount of cumulative experience not only in trading, but in writing financial systems. We filled in each others' knowledge gaps perfectly. They took the lead on the backend systems and I did the rest.
Cue months of finance training, seemingly endless numbers of commits (over 5000 between just the frontend and API alone, and that's only half the system), and we quickly pushed out our first beta of BitMEX. It's our take on a derivatives exchange with a much more finance-slanted perspective than our peers. We did months of paper trading and went live November of last year (we're almost to our first birthday!). In that time, we've explored many of the very hard problems in finance (margin trading is incredibly complex), and many more in UX (in short: finance is hard, making it simple for users is even harder). And now we're trading nearly $10M/day and growing.
What's so great about today is that there are so many markets out there you can play on, and so much code to learn from. If you try Stockfighter, I commend you. I'd like to offer our simulation to the community - we run it at https://testnet.bitmex.com, and a really simple market making bot at https://github.com/BitMEX/market-maker. In about 20 minutes you can be up and running market making on our Testnet and trying out your strategies.
Yeah, it's not the most complex bot out there (and we forked it from an old MtGox bot). It pretty much does the minimum possible to quote both sides of a market. But it's a great building block for more complex strategies, and it has all the parts you need, including Websocket hookups. Come grab me on Freenode #bitmex-testnet if you want help with it.
I hope some of you give Stockfighter or our Testnet a try. Finance is a rabbit hole, but it's worth learning about.
(tl;dr: Moved to Hong Kong, met some finance pros, wrote a ton of code, and now am CTO of BitMEX, one of the top Bitcoin exchanges by volume. We have a simulation market, it's worth trying out)
Well that is what they are doing, say you want to sell you stock in apple because the new iPhone is crap. Well Apple doesn't pay dividends (i think Microsoft is the only tech stock that does) so the only way anybody wants to buy is if they think the stock will go up in the future, so you need to find somebody who wants Apple stock more than they want the money. Well what if you can't find somebody right now? That is where the makers come in and buy the stock for a short while, hoping that they are then able to sell it shortly to somebody else for a profit.
I think what is confusing you is that a grocery have one price for an item and very rarely sold out, whereas as a stockmarket is more like a market: you are dealing with multiple sellers and not guaranteed to be able to find what you want and the price may depend on where you get it.
By money, do you mean IOUs? Who's handing out the IOUs during illiquid times? Certainly not the exchange, they have no interest in bearing risk of holding actual positions, they're merely providing a marketplace.
I saw this email today - sounds like some cool play to me.
There are both trading problems as well as lower level challenges. Seems like there is something for everyone. The idea that hits home is that it presents problems which you may not know about in domains and technology areas that are not ones that you work in so you get to flex yourself a bit in new ways.
A fun programming challenge is building an open source platform or some other software that scratches your own itches-- it's being a creator.
Solving arbitrary puzzles someone else has set in front of you, in order to get a job, is not a fun programming challenge.
It's one engineer saying "I bet I can come up with something this guy won't be able to do" and clueless companies and HR thinking they can "evaluate" programmers by "how well" they do on these quizzes.
They will miss out on the best programmers because the best ones value their time enough to create new things-- not solve "challenges".
Challenges are attempts to make you fail. They are not about learning about the person.
If you have self respect, you will use Straighter as a filter to exclude companies that are not worth working for (because if they use Starfighter, you can guarantee their working environment is not conducive to a career where you look forward to going to work every day.)
Starfighter should pivot into education. Then this work would be useful. Right now it's corrosive to programming as a profession.
Here's a thought: If you're playing Stockfighter simply because you want a job, you probably aren't the target audience for the game.
By that I mean, they have two different target audiences: companies who want to hire people, and separately, developers that want to play their game. When there is an overlap of those two groups where they can make a job connection, then great.
I don't think the goal of this is to make it so that anyone who wants a new job plays Stockfighter.
> Solving arbitrary puzzles someone else has set in front of you, in order to get a job, is not a fun programming challenge.
Except this is how actual jobs work. Real business value-creating requirements tend to resemble an arbitrary puzzle much more than something fun to scratch your own itches.
You may be right that Stockfighter won't attract the best programmers. It's not trying to. Those programmers don't need and aren't hired via any sort of challenge system anyway. What Stockfighter is trying to attract and identify are programmers capable of being plugged in to a real-world business application to create value.
>Real business value-creating requirements tend to resemble an arbitrary puzzle
Actually they don't. I say this with 25 years experience as an engineer and a couple patents and a long history of solving very hard problems. The hard problems are rare, the trick questions and puzzle type problems virtually never exist in business and are useless for evaluating programmers.
Programming is about a lot more than solving puzzles.
Why not simply give programmers the NYT and have them solve the crossword puzzle? That's what this is the equivalent of.
It only addresses one aspect of programming and actually a very small part.
Thinking about systems. Communicating with code that others can read, and communicating about code in a way that others understand, these are skills not addressed here. Maintenance and reading others code, also not addressed with trick questions.
I really have no problem with them making programming based games-- if the games are fun, great.
My objection is only with using this as some sort of recruiting tool. It shows a lack of understanding of what programming is, and it optimizes for exactly the kind of recruiting tactic that has effectively destroyed the software development profession during the past 20 years-- it's seeking code monkeys, not software developers.
The reason software is broken is the current model is "get a biz guy to tell code monkeys what to do, hire the fastest code monkeys and pay them as little as possible using the threat of outsourcing to indian code monkeys to keep the min line."
The ability to find a faster code monkey (as measured by their score on a coding game or hackerrank or any other arbitrary score) eliminates the messy need to have the hiring team actually understand software development enough to hire developers.
A better solution-- one that would be cheaper and more efficient -- is to hire more expensive software developers and have them lead other more junior software developers to develop software that actually solves business problems.
It's shocking how rare this is.
Straighter is enabling the problem, not trying to solve it.
"Monkey" and "indian" indicate a poisonous attitude, even at 50 years experience. I've dealt with people from India and Brazil, and they are not monkeys. Health and happiness to you anyway.
The term is not monkey but "code monkey", a standard term to refer to how some businesses treat software devs, not how the devs themselves are. And businesses like that are everywhere, including India.
I believe the etymology of the term comes from the infinite monkey theorem.
You have no trouble imagining a site like Project Euler funding itself from ads, right?
And you can imagine that virtually nobody who looks at the ads on an ad-funded site actually buys things because of the ads, right? The ads have nothing to do with why people come to the site.
Just mentally substitute "recruiting" for "ads" here.
I highly doubt I'll get a job out of Starfighter, if only because any good compensation (150K+) is probably gonna require a ton of work. (Especially being outside of SV, since such salaries are really top end so the demands are too.) I'm comfy enough doing little consulting here and there. The most I can see getting out of Starfighter professionally is the promise of 10-15 minutes with Patrick and Thomas (I think that's the prize if you win?).
That said, I'm immensely looking forward to this and have already spent dozens of hours reading up on things and over a hundred bucks on books. (My main decision right now is if I use a higher level language like F# or Haskell to be elegant, or use something lower like Rust for speed. Seeing as they didn't mention any low latency interfaces in this post, I'm guessing high level it is.)
Literally anyone in the industry can get me on Skype, without ever playing Starfighter. I'm hoping that remains true forever, and would be sad if it became untrue.
To the very limited extent you're competing on speed, in Chapter 1 you're competing with a) a Ruby script which b) is forced to sleep for roughly 80~90% of the simulated trading day. You don't exactly need to strive hard to win the race.
Common lisp actually has that sorta switch built-in: you can choose to have the compiler optimize for safety-checks, debugability, etc or speed. Normally that ability doesn't really matter, but if you don't know your problem domain well enough, that ability might be useful.
But - as someone that _loved_ Microcorruption and followed StarFighter from day one, stalking the people on Twitter as to not miss the launch: That's not for me.
The game right now - as this article, this mail shows - requires you to learn about finance systems before you even start playing the game. It's a game where the rule book is - from this pov - too long and already uninteresting.
Good luck, I'll sit this one out and would love to see a future expansion in different directions. Or another Microcorruption, Stripe CTF etc.