I couldn't agree more with your final thought. I assumed there may only be one or two diverted flights in a day, and certainly a handful more during winter weather, etc.
I clearly remember when I finally let my initial prototype run on a laptop for a day and ignored it, an old HP laptop just chugging away parsing flight plan data, plugged in and running too hot in my office. By the next day I had > 20 diversions, with no notable weather, and assumed I had a bug in my logic.
Lots more to do here, including adding some weather details and someday predictive analytics. Thanks again for checking it out and the feedback.
Finally getting around to sharing my long overdue side project that I had started back in Jan 2021.
The data processing is handled by two EC2's running Java to parse the FAA's flight plan feed. Peak midday that's running about ~65msgs/sec, a reduced and manageable number because I'm filtering for major airlines only (no GA, no bizjet, etc). The cleaned-up and filtered flight plan data heads to both Postgres for storage as well as SQS.
Consuming the SQS messages is a Python app that's crunching flight plans and confirming each flight is still heading to it's original destination (thanks redis!). If the destination airport changes, the flight gets flagged and some add'l business logic runs.
Ultimately you get the frontend (Django) presented here with a sprinkling of terrible JS to get the map and some timeline data.
I've been hacking with the FAA's SWIM data feeds for a few years now and hope to start sharing more of the technical details. Happy to answer any questions or hear some feedback!
Pretty cool. For those wondering the source of the data, it is likely ASDE-X/ASSC streaming in from the FAA's STDDS SWIM feed[0]. The terminal and ground data is especially cool and not as common as the usual flight tracking you see of en-route aircraft.
I'm using the STDDS feed to build a rough "go-around detector" in AWS for approaching aircraft by monitoring each approaching aircraft glide slope and dispatching a SNS notification "if currAlt > prevAlt".
If anyone has any questions about this stuff feel free to reach out, my email is in my profile. The easy part (IMHO) is slurping in the data which only requires a couple small EC2's and RDS, ~$125/mo. The hard part is often the presentation and making demos like these fun and shareable.
I don't think I've seen FlightAware showing aircraft tracks on the ground.
They do show taxi times for arriving/departing aircraft and I wonder if they are stitching that data together using the STDDS feed.
To give you an example, for an arriving aircraft you would get an alert of "wheels down", which is as literal as it sounds: a timestamped event when the aircraft made contact with the ground. In addition to ground position events (allowing the tracking you see here), you would also receive an alert when the aircraft left the tracked movement surface, i.e. transitioned to the gate/ramp area and no longer on an active airport surface.
(Pretty sure I have that correct, it's been awhile since I got back to my STDDS-powered projects. If anybody familiar with STDDS wants to chime in, please do!)
For the curious, all these "alerts" are JMS messages with XML payloads. Pretty much non-stop stream of data, incredibly fun if you're a developer+av geek. For FlightAware-style tracking without using any ADS-B you'll want to subscribe to the SFDPS feed. You'll get position updates every 60 sec for all airborne flights in the US, which last week amounted to ~40GB in my Postgres DB. You can request higher resolution reporting which is every 12s.
FA doesn't always show ground data (they must do some sort of filtering), but it is available via ADS-B. If you visit sites like www.adsbexchange.com or look at a receiver's raw data, you'll see ground data.
Good question. I haven't stumbled upon any, but it's possible they are out there.
The FAA offers a sample client. Depending on your level of skill, the hardest part might be the onboarding with the FAA which includes establishing a site-to-site VPN between your network and theirs. Once that is setup and some other networking is configured, you can use their Java demo client with the credentials they provide to get the hang of it.
The source of data is an event queue, powered by Solace, using Java's JMS standard. You can write a little bit of a Java to parse the incoming messages to get to the XML payload. Honestly I found that part to be the easiest and the Linux networking parts (as part of the VPN) to be the most difficult.
I wrote up a bit of a guide here, it's a little rough and high-level but I hope it helps clarify things a bit if you feel like working with these feeds/data.
HN'ers might appreciate learning the FAA has a developer-friendly program called SWIM [0] to help facilitate getting realtime FAA data (flight plans, weather data, etc) in to the hands of coders. I recently attended the SWIM Developer Workshop at the Volpe Center (near MIT) and really enjoyed it.
There's quite a bit to the SWIM program. The onboarding can be a bit tricky though and you have to know your way around site-to-site VPNs. I recently completed the onboarding process and am receiving live flight plan data -- proposed flights, en route, completed etc.
Would be curious to know if there are others on HN that are working with SWIM data or if there are companies that are hiring in this space.
I'd be interested in hearing your thoughts on the differences between using SWIM vs a commercial product like FlightAware. From my cursory glance at SWIM, it seems like it's probably better for getting flight plans and weather info, but FlightAware may be more useful for getting in-flight status. Does that seem consistent with your experience?
Makes sense. I know FlightAware operates their own network of ADS-B receivers, which augments the FAA data. I should look into whether or not that changes things, but I imagine the SWIM data feed doesn't have many holes.
In 2012, YC invested in a company InstallMonetizer[0] which, from my understanding, helps align software products with bundling other installers for additional revenue.
Impossible to have so many investments and not have a few bad apples. Still surprised PG did not distance himself further from installmonetizer, otoh props for standing by his investment and doing damage control for them at the expense of the YC cachet.
And Airbnb was spamming Craigslist posters (https://growthhackers.com/companies/airbnb/ - but of course selling cereals is the anecdote preferred by storytellers), and I have freelancer friends who got totally screwed by YC companies ("We can't pay for your work right now but we'll hire you and give you tons of equity right after demo day!" ... only to never respond to their emails ever again. Yes, the few of my friends to whom this happened could have been less naive, but still shitty).
The vast majority (but not all) of startups do dubious things. That's to be expected in an environment that glorifies "breaking things" and worrying later if what they're doing is legal or not. Being YC affiliated does not change that in the slightest bit.
People disliked them because the founders acted like obnoxious fratbros, but the specific claims of wrongdoing were over blackhat SEO ( https://news.ycombinator.com/item?id=6956658 ) which ended up getting them penalised by Google.
There's a wide variety of these "Pay-Per-Install" (PPI) type services, all of which profit by installing some form of malware on your machine. Sketchier services pay more, but also install stealthily rather than asking. Any kind is a pretty disturbing way to profit from your users. They're the kind of things you see script kiddies deploy to a small botnet. Not something you expect from a legitimate company.
I agree that this is a tactic that preys on the less tech savvy folks, and share your continued astonishment that reputable companies would engage in these drive-by toolbars/extensions that are bundled in installers.
YC has invested[0] in a company called InstallMonitizer[1] that appears to help developers and advertisers connect in the pay-per-install marketplace.
Sadly it does't seem like a practice that will go away any time soon. I'd like to do some digging around on developer forums and see if any folks have shared their experience and would be able to comment on the amount of extra revenue that they see from such programs.
Migrations[0] are being included in Django 1.7 which is currently under development. For those unaware, Andrew Godwin, the author of South, ran a successful Kickstarter campaign[1] the fund this effort to incorporate migrations in Django core.
Yes, this is surprising. Twilio has a reputation of being awfully transparent -- they had a billing systems issue back in July and handled it quite well, IIRC. They also do a great job with their developer evangelist program.
I wonder if someone has created a pricing matrix/table that updates and compares the SMS/voice API providers. For example, I know Plivo is slightly cheaper for SMS, domestically (USA). A price comparison table might make a good holiday project :)
Although I understand this wouldn't help your case. As you pointed out, the SMS provider would have already increased rates by the time you could be notified of increases.
Thanks again for providing all of the detailed comments concerning the advanced notifications for pricing changes. Your points are sound, and I understand that especially for customers with high volume use cases, relying on the pricing CSV file would not remove the costs of sudden price changes and would pose an the obstacle to the pro-active re-routing of traffic.
I've shared all of your comments with our product team, and I'll let you know as soon as I have more information concerning potential pricing change notifications. This would be a major change in the way that we price our Global SMS service, but I hope to be able to pass along some comments from our team by the beginning of next week.
We appreciate your thoughts about our SMS service and our pricing. I'll provide an update as soon as I can."
I should hope not. I'm trying to think of a situation where a developer wants to document a process (ANY process. In this case: building a profitable product) and it would be considered "bad".
> That's why success-story books sell, but you don't see a lot of "will-try-to-do" books.
This is a blog, not a book. Will-try-to-do blog posts are incredibly healthy exercises for the author/do'er and usually provide helpful content for the readers (if the author follows through and documents his or her thought process).
I'm not sure what point you're trying to make here, and (to me) your comments and tone sound dismissive. This fellow has nothing to sell -- it's an exciting endeavor that the author wants to document.
Tell you what - I'll get back to the topic & blog in 1 year time. I'll chat you up on HN and we'll redo the discussion. I'm willing to admit my mistake, but if they jakub ends this project in a-la-tim-ferris "book about selling books", I'll call my hunch correct - that he just played the community for self benefit.
If the project is left-for-dead, I'll consider my second statement about drawing attention too soon correct.
On the other hand, if the project is showing progress and does a fair writing - I'll gladly apologize to jakubgarfield and to you.
I clearly remember when I finally let my initial prototype run on a laptop for a day and ignored it, an old HP laptop just chugging away parsing flight plan data, plugged in and running too hot in my office. By the next day I had > 20 diversions, with no notable weather, and assumed I had a bug in my logic.
Lots more to do here, including adding some weather details and someday predictive analytics. Thanks again for checking it out and the feedback.