It's a shame. I had high hopes at the beginning that S4TF - and the investment in Swift from Google - would help Swift break out of the iOS ghetto and cement it as a mainstream language.
Swift's a delightful language to use. It has a lot of the nice things about Rust's type system, but is a heck of a lot easier to use at the expense of a bit of performance. For a lot of use cases, I think this is a great value proposition. ML/data science is a great example where practitioners could benefit from a more grown-up type system than what Python has on offer, but would still be able to keep low level details at arms length.
I think Serverless would be another ideal use-case for Swift, where the productivity, clarity and correctness tools it offers would be a huge benefit.
Some very interesting things came out of the S4TF project - like the work on autodiff, and python interop. It's a shame the project never really seemed to get legs, it seems like it just languished for years with no clear direction.
These days I do most of my work in Rust, and I'm happy to do so because the tooling, community, and ecosystem is really amazing. But there are a lot of language concepts and features I miss from swift. I guess it goes to show that the governance and availability of a language have a lot more to do with adoption than the merits of the language itself.
I love all languages in the ML/Haskell tradition, but I think Julia would have been a better fit because it's dynamic yet efficient, and because it has a really decent probability/statistics/ML ecosystem already. Long term, I think it's the best replacement we have for Python in the ML world.
Python has exceptional libraries but, as a language, it's a bit dated on several fronts. This has an impact on library design. In Python, ML libraries are huge monoliths and they depend on a lot of code written in other languages. They are really hard to understand or modify.
In Julia, things are really small and composable. For example, you have a probabilistic programming library like Turing and a differentiable programming one like Flux, and it's trivial to implement some Bayesian neural networks. Same applies to many other things. It's small and beautiful, but it needs more manpower to compete against Python.
> In Julia, things are really small and composable. For example, you have a probabilistic programming library like Turing and a differentiable programming one like Flux, and it's trivial to implement some Bayesian neural networks.
I was taken aback when looking at Turing for Bayesian modelling that the distributions were just the standard distributions found in the Distributions package! In Python, every Bayesian framework has its own implementation of everything from distributions to log probabilities, but it all composes in Julia.
> It's small and beautiful, but it needs more manpower to compete against Python.
Agreed. Docs of major projects are still incomplete or non-existent, there's a lot of projects that have been nigh abandoned (cough naive bayes cough), and the composability of Julia coupled without strong leadership leads to an ecosystem of overlapping functionality. Still, I hope by leveraging PyCall, Julia can overcome some of these issues.
You mean that, in Julia, we the users have to "compose" our own implementations of models (e.g. log probabilities), as opposed to using the already-made ones in Python?
More like... most of the scientific Python stack has settled on a third party library, NumPy, to provide support for numerical computing. That includes providing data types like arrays. Tensorflow doesn't use NumPy's data types, so if you want to use Tensorflow Probability, you need to convert everything you've done in NumPy (or things that are built on top of NumPy, like Pandas) to TensorFlow, and use TensorFlow's functions/methods to manipulate them.
(The same is true if you use something that's built on top of Stan or JAGS or BUGS or something else.)
In Julia's Turing.jl, everything is built around data structures that are first-class parts of Julia, so there's no need to have special Turing.jl versions of, say, probability distributions.
yes, and if you go to that link, you can see its experimental, for TF 2.4 _and_ the notebook on github was created Aug 3, 2020. So that is relatively new and most likely has a ways to go before further adoption. Its good that option exists, but doesn't seem very backwards compatible. That doesn't negate the points made in the parent.
It will mature before the Julia ecosystem mature so the value proposition of Julia is out weighted by it's lack of ecosystem.
Had they transparant interoperability with Python like Kotlin is for Java, I would consider it.
They could at least rewrite their vm to use the truffle framework from graalvm so that they would become interoperable with Python on graal which in a few years will mature.
If they weren't already-made in Julia yes (and a library didn't do the work of composing it and presenting as a monolith, since pretty much all large frameworks like Flux and DiffEq are also made this way). Although you're maybe trying to imply that everything that anyone needs is already made in python, I occasionally end up finding small issues between my use case and what's immediately available and I have to twist my problem into a stitch of "kinda what I need" pieces (like having to change my problem into a vectorization problem, a long chain of corrections using pandas to fit an interface and then some weird logic to transform the table into a CSV in memory to use postgres' COPY or UPSERT because pandas to_sql is not flexible enough and iterating and inserting is too slow). Sure I always achieve the result using already-made stuff, but the glue logic ends up taking more time and honestly ends up quite hard to maintain for other people (or even myself after a few months) after all it isn't my algorithm logic but some adhoc puzzle I just solved.
In Julia, if the DataFrame library is missing something I can just loop like an array and have a method that works just as well as if the library provided it, the CSV, DB and table processing libraries all use the same conventions so if library "A" solves my issue I'm not forced to use the same library "A" serialization method. Basically instead of twisting the logic to what I'm given, I just fill in the blanks. Sure Julia has way more blanks, but Julia also has half of the age of some of the Python's library I use, it's more about the maturity of the community than anything to do with the design choices of the language and I can only hope it gets better and better with time.
Julia gets compared a lot to Python but truthfully it strikes me as more like a replacement for FORTRAN, MATLAB, and R. The ergonomics of the language seem good but documentation is poor and as a community it's narrowly scoped into the ML/numerical processing world.
If all you do is ML/numerical processing, against data that's already been cleaned up, I bet it's really great tho.
The comparison is natural because Python is a general purpose language which happens to be used for data science, the same can be said about Julia. The others languages mentioned are not really that great for general purpose programming.
But I agree with you that in the short term it is a replacement for Fortran, Matlab and R. But a new language has to start somewhere. Attacking the niche filled by these language is natural as these are old, dated and ripe for disruption. The key is to gain some critical momentum and user base. That will carry the language to other domains.
> Python has exceptional libraries but, as a language, it's a bit dated on several fronts.
I've been using Python for ML for the last 3 years and I've never felt this way. It might be that I'm not all about the hip new languages, but I don't really see the benefit of making Python more ML/Haskell-ish.
The ML use case for Python is roughly as follows: you design, train, and evaluate models. Then, if something is decent enough to use in production, you switch over your application to load and use that instead. I don't really see where Haskell or any language from the ML family can improve in that process.
Sure, the code that you used to implement your model may improve slightly, but I don't see that code improving significantly. The fruit of your labor is usually a protobuf file (encoding the TensorFlow graph) or whatever your framework uses to encode the model you built. The code actually surrounding it is very minimal for most use cases.
> In Julia, things are really small and composable. For example, you have a probabilistic programming library like Turing and a differentiable programming one like Flux, and it's trivial to implement some Bayesian neural networks.
There's nothing stopping you from composing things in Python. But it's simply not a goal of 99% of ML libraries to be composable with other libraries. You're probably never gonna run TensorFlow and PyTorch in the same application (trust me, I've tried, it was a nightmare) and I don't see why you would compose a TensorFlow model with a PyTorch model without incurring tons of overhead in your application around gluing these two things.
> There's nothing stopping you from composing things in Python.
There is, and you pointed it out yourself:
> I don't see why you would compose a TensorFlow model with a PyTorch model without incurring tons of overhead in your application around gluing these two things.
This is where Julia is profoundly different. Reusing stuff from different libraries is trivial compared to Python. You could take an activation function made for one libraries use it without any modification or wrapping in another ML library.
These is where Julia will win long term. If you look at the ML libraries in Julia they are tiny. The Python side of things require gargantuan effort because things are not composable. The wheel is reinvented over and over again.
>there's nothing stopping you from composing things in Python.
In Julia you can compose a custom distribution, with a bayesian model with an ODE with a neural network with unit number types with custom Julia written CUDA kernels and multithreading.
Edit: That are not designed specifically to work with each other
Can python even hope to do a fraction of that, still be fast and differentiate through everything?
I'm fairly certain everything you said is possible except for custom CUDA kernels in pure Python. You'd have to write the kernel in C++ and use it in your TensorFlow/PyTorch code. [0][1]
> It's definitely not possible. These have to be rewritten with the specific autodiff /ML framework in mind.
I don't understand why it's not possible. You're asking if it's possible in the language. I don't see anything stopping you from, as you say, writing your own framework, or simply doing it in TF. Perhaps my ignorance of Julia is showing :)
Admittedly it is a bit hard to see why Julia is so different in this area if you have not spent some with it. It took me a bit time as well to see.
But there really is a profound difference. The thing is that Python is really just a language to rearrange networks of nodes written in C++. A node written for one ML framework will be incompatible with a node written in another language. If you want them to work together you need to spend a lot of effort gluing them together, and most likely the performance benefits are lost.
This is very different in Julia because you are using pure Julia code. What is specialized C++ nodes in Python, are for the most part just plain functions in Julia. There is nothing special about them. That is why they can be used anywhere.
In Python frameworks each of these C++ nodes need to contain information about how auto-differentiation is done. That is why they are tailor made for one specific ML library.
In Julia the whole automatic differentiation stuff is perform code transformations on regular Julia code.
Are you familiar with LISP, and the idea of code as data? That is what we are talking about here. A regular syntax graph of regular Julia code gets manipulated to do autodiff. You are not organizing C++ nodes in a network.
> A regular syntax graph of regular Julia code gets manipulated to do autodiff. You are not organizing C++ nodes in a network.
That does sound very different than Python. Admittedly, I haven't used Julia, because I never had to for professional purposes. For personal projects, I use PyTorch, because I find it easy to use, intuitive, and I have a lot of experience with it. Maybe one day I will be forced to use Julia and will appreciate what it brings :)
That was a great explanation that makes me appreciate Julia more, thanks.
I don’t much use Julia, but it is a beautiful language. I started a tiny side project for using Julia for non-numerical things like querying SPARQL endpoints, text processing, etc. Julia could be a universal, use for almost everything language.
>I don't understand why it's not possible. You're asking if it's possible in the language. I don't see anything stopping you from, as you say, writing your own framework, or simply doing it in TF. Perhaps my ignorance of Julia is showing :)
It's not that it's not possible. Yeah, you could always "write your own framework".
It's that it's not readily available by the language, ready-made to compose, without resorting to external libs on C/C++/Fortran, and so on.
So, "not possible" not in the sense that "it's not possible to write a program that solves the halting problem". More like "it's not possible to write a web app in assembly". Which means, yeah, it's possible, technically, but you wouldn't like it, it would be full of holes, a pain to maintain, and not a good time investment.
Let's say you write some code that handles numbers differently, say a library that implements quaternion math. For doing graphics manipulations. Then you try to use it with tensorflow maybe for some ai-driven optimization. Is it going to work? Probably not out of the box. You will have to do some munging of quaternion data type to shove them into tensorflow tensors. Do the same thing in Julia with Julia's flux, it probably will, and maybe even with the gpu That's the difference.
If you're rewriting everything to get the behavior rather than just hooking together preexisting pieces, that's not what's generally meant by composition.
They key concept that allows for this high degree of composability is multiple dispatch; it’s like if you were able to define trait for any function on any parameter type or to overwrite any existing function to specialise/generalise on one or more parameter types. Ie instead of wrapping different libraries to glue them together to work on your data, you update existing functions to accept your data, ie you blend your thing into existing things instead of wrapping. And once blended your data works not only with that library but also with libraries that use this library without extra effort and with very terse final code.
The Julia approach gives quicker development. Some of these ML libraries such as Flux are so small that almost anyone can read the code and learn how it works and make modifications. It is not for anyone to jump into TensorFlow.
The irony here is that tiny Julia libraries give the same power as much larger Python libraries which require highly specialized and trained developers to evolve and maintain.
With Julia it is much easier to grow the eco system because you can make lots of relatively small packages which can then be combined in almost any way.
The challenge for the Julia community today is really to make people new to the environment aware of this.
I have encountered people who thought Flux couldn't do anything because it was so small. He was so used to the huge monolithic libraries in Python, that it did not occur to him that adding something like an new activation function is literally one line of code. He is used to that requiring adding various C++ nodes. Wrap those up and God knows what more steps you need when you extend something like TensorFlow.
I don't think most people realize how powerful Julia is. They are not used to being able to combine libraries the way you can do in Julia. A big part of the effort for the Julia community will simply be to write more tutorial and introduction which better introduce beginners to these kinds of capabilities. Once you know Julia it is pretty obvious.
But grab people's attention and make them realize Julia could be a solution to their problem, I think one needs a lot of sort of shallow and quick intros to these kinds of things.
> The challenge for the Julia community today is really to make people new to the environment aware of this.
^This. As we're seeing here in the discussion thread Python folks don't really realize what they're missing when we're talking about composability of of libraries in Julia - it's kind of difficult to explain the impact on productivity without people actually trying it out. I think Julia and Flux are getting to the point where they're quite useable and comparable with PyTorch. Also some of the Neural ODE stuff and differentiable programming in SciML seems just a lot easier to implement than it would be in Python.
From the point of view of a DL researcher, you’re able to easily implement your own custom layers. From the point of view of a framework developer, you get much faster development time. From the point of a more normal user, you hopefully get more features faster because the jobs of the framework developer and DL researcher have been made easier.
> I've been using Python for ML for the last 3 years and I've never felt this way. It might be that I'm not all about the hip new languages, but I don't really see the benefit of making Python more ML/Haskell-ish.
Julia is not ML-ish. In fact, at first blush, Julia reads very similarly to Python. You even get list comprehensions in Julia!
> Sure, the code that you used to implement your model may improve slightly, but I don't see that code improving significantly. The fruit of your labor is usually a protobuf file (encoding the TensorFlow graph) or whatever your framework uses to encode the model you built. The code actually surrounding it is very minimal for most use cases.
It's really about breaking down the compositional boundaries here. Upthread, someone talked about how they could play around with Bayesian Nets by simply mixing Flux.jl (the differential programming library) with Turing.jl (the Bayesian programming library). Mixing Tensorflow/PyTorch with PyStan, for example, can be a nightmare. That's also why there's so many implementations of probabilistic programming frameworks (Edward, PyMC3, PyStan, Pyro, etc); they all use different underlying libraries. In Julia they just all compose together.
> I don't see why you would compose a TensorFlow model with a PyTorch model without incurring tons of overhead in your application around gluing these two things
I find the change to be illuminating. Before I started spending more time with Julia, I would often do derivations on paper for more experimental work, and then implement it using TF/PyTorch in Python from scratch, reading other code where I could for some help. In Julia I can import a library and I'm ready to go. It feels just like working with math itself.
Julia also lets you compose long trains of operators together. That also helps when I'm doing long sessions in the REPL exploring data. It lets me define a few functions in a file, import the file, and just pipe (Julia has a pipe operator which is just function application) output between functions to plot data or calculate errors.
Moreover Julia is a lot more performant for REPL work than Python. In Python I'll usually work with a subset of a dataset to get an idea for it, then run a file with code to process the entire dataset. In Julia I can often prototype and run the algorithm in the REPL itself.
I also want to stress that Julia is quite a bit faster for REPL development, both in terms of raw speed
I largely agree, Julia is such a cool language and had so much potential. It definitely surprised me when they went with Swift instead, but realizing that Chris Lattner worked at Google at the time explained a lot. Unfortunately, every time I try to get into Julia, it just feels awkward coming from Python and a bit like stepping back in time.
The stupidest, (stupidest in the sense that I really wish they didn't bother me, because they're silly things!), things that bother me are:
1. The 'end' keyword. It's everywhere! Loops, if-else statements, function bodies. I mean I understand why it's helpful for parsing and providing more info to the compiler, but honestly, I would've rather we just stuck with curly braces '{}'! At least it's fewer characters to type and it just feels less crowded on screen while reading. It feels like such a petty complaint, but honestly, it feels like I'm writing Pascal or Matlab all over again. Which leads me to my second point.
2. The default choice of 1 based indexing. I'm not going to go into it, because plenty of people before me have beat this dead horse already[1][2][3], but I can't help but be saddened by this choice and its implications. It's important to acknowledge the fact that Julia started as a competitor to Matlab and Octav, so it makes sense from that perspective. However, it could have been a much more general purpose language, with huge performance benefits over popular interpreted languages like Python, JS, and Ruby. It could have been a unifying force in the scientific computing community, bridging the gap between R and Python users with a greenfield approach that would have been a force to be reckoned with. Instead, rightly or not, it's viewed largely as 'just' a matlab replacement.
Now, regardless of whether 1 or 0 based indexing is truly 'better' or the 'end' keyword is no big deal, the reality is that there's a huge demographic of potential users that won't buy into Julia, because it doesn't quite feel as ergonomic as Python/JS/Ruby and won't take it seriously as a general purpose language, because it looks/feels like Matlab and they only use 'real' programming languages. Again, I'm not saying this is right, but it is the reality we're faced with and it just feels like a huge missed opportunity and bums me out.
Language aesthetics do matter. Broadly it seems like we've accepted that "c like" syntax - including brace-delimited blocks and zero-based indexing - should be the norm for programming languages. Any language which goes in a different direction should have a very strong reason to do so, because any deviation will be an obstacle for adoption.
I share your frustration with the `end` keyword - it's just needlessly verbose, and for something used so frequently it makes sense to be as terse as possible.
I have some similar quibbles with Rust syntax: I know it's a minor issue, but I'm really disappointed that snake_case was adopted. It's just ergonomically inferior to camelCase in every measure. It's one more character to type for every word, and on top of that, on a US keyboard, the underscore character is a pinky key way far away from the center of the keyboard. Making this one of the most frequently typed characters in the language makes no sense.
snake_case is a lot more readable than CamelCase, which is a huge benefit ergonomically. The keyboard layout is no big deal, one can easily change it or use shortcut-based autocomplete. Rust does use CamelCase for type identifiers, trait identifiers and enum constructors, and the contrast with snake_case also aids readability.
I disagree that it's "a lot" more readable. I have read a lot of camelCase code in my lifetime, and I can count on zero hands the number of times I ever had an issue parsing code due to the use of camelCase.
Keyboard remapping seems like an extreme solution, and I don't want to train my fingers in such a way that when I sit down at a different workstation that doesn't have my .vimrc I can't type rust anymore.
You don't need snake_case for contrast. You can use lowerCamel and UpperCamel to denote the same levels of significance. SCREAMING_SNAKE is fine for constants because they don't get used all that often, but rust maps the hardest-to-type case to the most-frequently-used, which in my opinion is an ergonomic failure.
It really depends on your naming conventions. If you consistently treat acronyms in the same way as regular words (i.e. "XmlReader", "IoStream" etc), then there's no practical difference with "xml_reader" and "io_stream".
Can you give an example of where you find this "terrible?" I have never had an issue with it, and even if you don't like it aesthetically, I think the fact that you don't like looking at certain names (a subjective complaint) is objectively easier to overcome than being required to type one of the least ergonomic characters on the keyboard over and over, which is a matter of physical reality.
So would you seriously suggest that programming languages should be optimized for extremely rare, niche keyboards rather than the standard that ships with virtually every laptop which nearly 100% of coders will be using?
`end` marks of blocks of code much more clearly than curly braces. It is also requires fewer keyboard taps. To type {} requires holding down four keys in total (shift-[ twice). end is just three key strokes.
But more importantly is saves curly braces for other uses where it is more needed.
Typing braces is basically a gesture. You hold the shift key with left pinky, and roll right ring finger pinky over open and close brace, then tap back arrow and return to get you inside the block. End is three distinct key presses.
Also braces have a nice symmetry, which is also convenient for parsers and tooling to count opens and closes.
I also think "end" is just more visually noisy. Braces are a symbol, so it's easy to filter them out when you're looking at code, but a trail of "end"s takes up more real-estate than is semantically justified.
It seems odd to say "we just write" for some arbitrary choice with its own tradeoffs; {} mean a set in Python or a literal array initialiser in C# or a scriptblock in PowerShell or a dfn in Dyalog APL or a JSON dictionary, etc. With only a limited set of whatever symbols happened to end up on a US ASCII keyboard 40+ years ago, there is heavy competition for them, it's why J broke the symmetry of pairs and went for two-character symbols, why PowerShell went for comparators like -gt and -lt, why C# has => which is completely different from <=, why << is not a double application of less-than but is bit shifting ...
To act like there's an objectively better use for {} is to miss how many tradeoffs there are, and how many things people want symbols for out of a standard keyboard.
< and > are comparison operators for checking if something is less than or greater than something else.
This is okay in a static language where there are special slots for a type where only a restricted subset of things can happen. You'd know at the parser level that the < and > are referring to the type meaning or the operator meaning.
In julia, types are values and values can live in types, and arbtrary operations can happen in a type.
E.g. I can write Tuple{1 < 2} which just becomes the type Tuple{true}, i.e. a Tuple Type with paramemter True.
Or you could just not use them for type parameters. That's also a solution for what it's worth. I think the curly braces look better as type params anyways imo.
That's not a particularly convincing blogpost - for example you don't need square brackets for indexing, one of Ken Iverson's changes going from APL to J was to see indexing as just another function application and remove square bracket syntax from it. Grouping arithmetic expression is worsened by complex precedence rules, APL always evaluating right-to-left and all operators having the same precendence changes how that works as well (even though it does have grouping parens too). Then complaining that "<>" are used in mathematical expression, but not being annoyed by "main()" in a function declaration or the "?" and ":" used in the complex type expression?
Seems to me that "end" is one of the worse available choices; if we're going for keywords, why not make them "endif", "endfor", "endwhile" (or "wend") which makes it explicitly clear what each of:
Re: 0-indexing vs 1-indexing. If you use 0-indexing, you turn off a lot of non-engineering scientific programmers. My personal experience is that 0-indexing is better for more engineering applications, while 1-indexing is better for math. I'm a weirdo in that I don't seem to mind either one though.
> you turn off a lot of non-engineering scientific programmers.
I think you mean "non-CS engineers". CS is a minuscule branch of engineering. Plenty of chemical, mechanical, civil (and so on) engineers had their whole education doing maths and programming with 1-indexing.
> I'm a weirdo in that I don't seem to mind either one though.
You are not, as an outsider this is one of the less appealing parts of practical CS, endless bickering about non-substantive issues which most of the time boil down to a matter of personal preference (see also tabs vs spaces, vim vs emacs, react vs vue, golang vs rust and on and on and on...)
I mean, Julia is flexible enough that it's pretty easy to implement arrays with different indexing schemes that have the same performance as built-in arrays: https://github.com/JuliaArrays/OffsetArrays.jl
Zero based indexing came from early requirements to address things in arrays using offsets.
Subsequent languages don’t have this issue, and when you think about it, it causes a weird disconnect: 1-based indexing corresponds exactly with how you think about and see elements in a collection. 0 based indexing-despite how comfortable one gets with it requires additional arithmetic.
It’s a relic from the past, and unless you’re doing the very specific thing of indexing via offset instead of position, there’s no reason to hang onto this anachronism.
Most math books are written in a way where 1 is the first element. So if you take math examples and translate to code, it works more naturally.
Also this is how you talk normally. You don't talk about the zeroth-column or zeroth-row in daily speech. You talk about first column and first row.
Only reason 0 based indexing make sense to me is because I began programming as a teenagers and was forced to get accustomed to it. But I remember struggling with it. Yes when working with memory, pointers etc it is more elegant. But if you are not, then I think 1-based indexing looks better.
As a trivial example, if you want to get the sum of the first n numbers, that's `sum(range(n+1))` in Python. Or if you want to get all the prime numbers <= n, find the triangular numbers, etc. In general you end up with a lot of `n+1`s and it's easy to lose track or miss some, and end up with a silent, non-crashing error that produces the wrong result because you're accidentally leaving out the last element of your input.
Linear algebra is the most practically consequential one - and suffice it to say, it’s not a coincidence that BLAS, LINPACK, and LAPACK were all written in one-based languages. Matrix indexing is one-based, and beyond the issue of translating notation from equations, IIRC there is even a slight performance difference in some cases.
I believe what it really comes down to is that, just as zero-based really is more natural for offsets (zero offset means the first position, great for pointers), one based really is more natural for counting [and _sets_] (when your index is n that means you have counted n elements up until now).
I think your pain points with Julia are real, but they don’t bother me too much. I mostly use Lisp languages, enjoy but am not so good with Haskell, so I am very flexible in syntax. I shared the dislike for 1 based indexing, but when I recently started evaluating Wolfram/One and Language, my limited experience with Julia and Octave made Wolfram’s use of 1 based indexing OK.
I love experimenting with programming languages but I don’t like recommending languages just as I don’t like recommending movies: I figure that everyone has their own tastes, and that is a good thing.
If it's just the syntax and muscle memory you are quibbling about, you can use python to write code and then transpile it to Julia. Link to github in my other comment in this article.
> I guess it goes to show that the governance and availability of a language have a lot more to do with adoption than the merits of the language itself.
Nearly every Google project eventually gets abandoned because the engineers that made it get promoted and move on to their next promotable projects. The root cause is the ongoing failure of Google's leaders to align employee incentives with the long-term interests of the company or human society.
Even if S4TF had become popular, I expect it still would become neglected and rot like most Google projects.
>I think Serverless would be another ideal use-case for Swift, where the productivity, clarity and correctness tools it offers would be a huge benefit.
Oh yes, I would love to have Swift framework for Firebase on server, not only for iOS. Its atrocity to write the server logic in NodeJS after making the user App in Swift.
Every time I switch from Swift to JS I deeply appreciate the beauty of Swift. On swift I do much less silly mistakes where the NodeJS code feels like up and running by some miracle and everything can collapse due to something I did but could't catch it before something horrible happens.
Could it be that you have more knowledge of Swift and don't care/want to invest in understanding Node.js/JavaScript? I really enjoy writing backend code in TypeScript for Node.js. But loath having to write Swift code for iOS. I know a big part is my unwillingness to invest time in Apple's ecosystem and properly learn Swift.
I used to work with web technologies but I got overwhelmed with the complexity of the tooling.
For example, TypeScript is nice but it's not a "real" language in a sense that you can write something in it and expect it to work when it's fed into the compiler or interpreter.
To use it, you need to set up an environment where all the moving parts are working in harmony and the code you write in Typescript is transcribed into the actual code that NodeJS and the browser will understand and that is JS. The debugging of unusual bugs and doing something non-conventional instantly becomes multiple times harder because you have layers and layer over the actual stuff that is executed which means you loose the browsers or runtimes debug capabilities since those would rarely tell anything useful about the pre-transcription code.
Sometime you have a library where someone a few years back tried to do something similar to your idea but never had a complete solution and stopped working on it and when you try to benefit from this work to build on top of it, you find out that you need to modify you working environment to support some spacial case of legacy code. You do that and it all falls apart, now you must choose to try to fix it or give up and restore your setup. It's just horrible.
The greatest thing about working with Swift on iOS is that starting something new is as easy as creating a new Word document so you can start working on your idea right away without worrying about the tooling and see where it goes. On the JS world, I used to be exhausted and loosing all my motivation by the time I have my environment ready to go. Templates and project starters tend to be no good because they are all opinionated that you must be creating a list or a blog app. They all need extra work for a good clean start and that clean start often is too customised to be used as a general use case template.
There are so many solutions for the same problem in the Web Tech world, each comes with its own shortcomings and taht's how someone else creates the next framework. On iOS it's easy, the answer is UIKit and SwiftUI if you feel more adventurous and bleeding edge.
I agree about TypeScript, however this paragraph also describes Swift to a T:
> Sometime you have a library where someone a few years back tried to do something similar to your idea but never had a complete solution and stopped working on it and when you try to benefit from this work to build on top of it, you find out that you need to modify you working environment to support some spacial case of legacy code. You do that and it all falls apart, now you must choose to try to fix it or give up and restore your setup. It's just horrible.
So far Swift code has not aged well, although it's gotten better during the last two years. The perpetual brokenness has moved on to Swift's tooling (binary stability, SwiftPM etc.) which wouldn't affect serverless.
Deno seems to address some of the shortcomings of Node.
I haven't used it on a "real" application yet, but the built in support for Typescript, formatter, linter etc. reduce the complexity of tooling, especially for server code.
Which is the main reason why although I like Anders's work, I keep using plain old JavaScript, hoping that one day browsers at least support WebIDL annotations natively.
They have good defaults and Xcode is a good IDE with full support of these frameworks, Swift is a good language with sensible conventions so you don't really need to learn that much.
When you don't know something you can start typing keywords and see what Xcode autocompletes for and read the explanation about that function from the documentation that is provided to you right there. It's not perfect, sometimes there's no documentation but it's not a big deal as you can look into source files or try it out to see how it behaves.
It's not the same because in JS you usually would use a large number of libraries with varying degree of documentation and different conventions. Also, you would loose autocomplete when you expect to receive something as a parameter instead of defining it in that closure.
The basic language API is not that large, the benefit of autocomplete discovery comes when you have autocomplete on custom types of a specific framework accompanied with proper documentation.
I’m sure NestJS, which sparked this thread, has very complete type defs, as does any major framework out there.
You seem intent on dismissing something you obviously haven’t tried in the past couple years; next time I’d suggest a simple “I don’t like it” instead to save us all time.
Sorry you feel that way, it feels like angular for the backend to me which is nice, it follows logical patterns, loading modules allows us to configure micro services from a single mono repo, decorators are nice, encourages use of RxJS which is great and it still allows access to the underlying express app if needs be.
Might not be to everyone’s tastes but it feels like a solid tool to me.
Yep, Vapor looks nice but I want to use Firestore, so I need a reliable integration of Firestore with Vapor and there isn't one. I rely on listening for live data entries and updates on the Firestore, processing those and putting them back.
I'm doing it on NodeJS currently and I hate it. I used to like JS but Swift showed me that there's more, there's beauty in the world.
The only thing I miss on Swift is async/await and that's coming.
Yeah. I think unable to merge back into Swift mainline really makes the adoption harder. I made some efforts to having PythonKit / swift-jupyter run smoother and it feels really good as a replacement for Python: https://liuliu.me/eyes/data-science-setup-with-swift-in-2021...
Luckily, I don't think Swift on other platforms is dead. SwiftNIO seems worked quite well as a project. Bazel support is solid. They still release Linux / Windows Swift toolchains.
Also shameless plug, I did implemented my own deep learning framework in Swift: https://libnnc.org/s4nnc/
I agree and am also disappointed. On the other hand, I really don't miss Swift outside of iOS because of Rust and Kotlin.
I wonder if Kotlin is too far from "arms length" from the low level details in your mind? Because other than that, I actually prefer the language over Swift, generally.
Kotlin is one of the very few languages to have a compiler plugin API, this allows Facebook researchers to make it auto-differentiable on demand
https://news.ycombinator.com/item?id=25128251
While people can argue swift could be Golang, it's built with focus on iOS ecosystem. You have to manually handle reference cycles. The language is quite complex as well, and compiler is among the slow ones. Copy-on-write is generally despised these days even in C++. Just to keep in mind that this may give space for a next expressive general purpose language that improves in these aspects, we can be optimistic.
> Is the Python type system improving or is it the type annotation system that's improving?
Both. (The type annotation system is deeply tied to the type system, since the latter is what is statically verified, but there are improvements both in what can be checked—the type system—and how that is expressed/annotated.)
It's important to understand that the type systems for Python are not really an inherent feature of Python, but the systems enforced by each of the static checkers (which tend to adopt each other's advancements and drive syntactic support into the core system of annotations, so there's a common baseline advancing with the individual systems). A recent advancement in one of those type systems would be pylance (actually pyright, since that's the typechecker for the pylance language server) support for higher-kinded type variables [0]. Mypy, I believe, has identified this as a thing to support, too.
An example of an improvement in annotations specifically to enable type system improvements that require supporting syntax improvement is PEP 646 support for Variadic Generics. [1] Even though it is in draft, pyright has already landed support for it for the type system improvements motivating it. [2]
Oh, okay, the problem here is that we have different understanding of the relevant meanings of “type” and therefore “type system”, which is unsurprising because the term has many different uses in programming.
I am using “type” in the standard computer science sense of statically enforced invariant and “type system” as the system of statically enforced invariants, distinct from annotations which are the syntax by which types are expressed.
You seem to be referring to “types” and “type system” to refer to something else, perhaps some aspect run-time behavior / “dynamic typing”. whatever it is, there's probably also some improvement in that, too, but given that the original concept of the uphtread comment about type system improvements was about type safety, which is what the system of types that are statically verified addresses, it's not really a germane issue.
There's multiple aspects to what a type system can entail.
There's dynamic/static typing which is what most of the thread is about. Python is dynamically typed , which makes it really easy to pickup and make stuff in, but you get issues like runtime errors/bugs since there's no inherent type checking other than attribute access failure, or explicit isinstance calls. To the best of my knowledge, there's no advancement here in Python since it's antithetical to the language design. It is strongly typed however which is great, but not really part of the discussion since it's being assessed in the thread on the virtues of static/dynamic typing and not weak/strong typing.
Type annotation helps partially here since you can statically check the types notated post facto but this is not the type system. The type system in a language is a contract by the compiler/interpreter to enforce type safety. Annotations have no contract for correctness of description or even guarantees of being checked.
This is why I'm trying to draw the distinction between the type system vs the type annotation system. The type system itself hasn't improved much/at all in years by design. The type annotation system has however improved significantly.
> Type annotation helps partially here since you can statically check the types notated post facto but this is not the type system.hatever a static type checker enforces is exactly a type system (and the only thing that is), in the standard CS sense.
The fact that the type checker isn't integrated with the compiler/interpreter is a different design choice from other systems, but it doesn't change what it is.
Effectively, mypy (and this was explictly what mypy was originally envisioned as, as is still reflected in its domain name) and other typecheckers for Python create separate statically typed languages with the same relation to Python as TypeScript has to JavaScript, except that source code in those languages is also valid Python source code with identical runtime behavior. That is, each typechecker creates a new language that is a statically typed strict subset of Python, with its own type system (most of which is common across the whole set of languages, but there are slight variations as each are ahead of the common baseline in different ways.)
> The type system in a language is a contract by the compiler/interpreter to enforce type safety
No, it's enforced by the typechecker. Now, in most languages implementations that have a type system, that's part of the compiler, or more rarely interpreter, but there is no inherent reason it needs to be.
> This is why I'm trying to draw the distinction between the type system vs the type annotation system.
Yes, you've invented artificial nonstandard terminology here that is clearly not the terminology that was being in the post you argued against, in order to make a really extended argument that vigorously exercises the fallacy of equivocation and badly misses the point of the post it sought to dispute.
To the extent there is a legitimate point here it boils down to “Strictly speaking, Python remains untyped, and what is advancing are the type systems of a family of statically typed languages built around Python, that are strict subsets of Python, that leverage feature built in to Python specifically to enable such languages, and which are developed in close collaboration with Python, in some cases sharing core team members.”
The lack of algebraic data types is what python gets criticisms about. That situation is improving, but I would have preferred match to be an expression rather than a statement.
I get that you're not particularly excited about gradual typing and the fact that type checkers are optional and live in a separate ecosystem from the interpreter/language runtime.
I actually like the fact that python code without type annotations continues to work as before and allows for easy transcription of ideas into testable code.
If a certain use case requires "forced" type checking, it shouldn't be hard to write a lint rule that disallows code from being committed if there are type checking errors, including missing type annotations.
The issue though is that Python's type annotation is not a substitute for type checking. Annotations are optional and can be inaccurate.
That's fine, this is part of the design of Python. However strict type systems are a benefit of other languages, that may not fit Python. Type annotation shouldn't be considered an alternative to a type system.
> I think Serverless would be another ideal use-case for Swift, where the productivity, clarity and correctness tools it offers would be a huge benefit.
I agree, but sadly none of the big cloud providers has any interest in pushing it - Google's got Go, AWS and Azure seem focused on Typescript.
I'm running Swift server side on Azure. You simply package it in a docker container and run it in an appservice. The only problem really is not having native blobstorage support (like AWS S3) but I wrote an opensource implementation that supports the basics at least. Im not going to plug the library here however coz its far from finished :)
Yes theoretically I think there is nothing stopping you from shipping Swift code on AWS. I am pretty sure you could do it in a lambda and you don't even need a docker container.
If you're talking about a container workflow, I'm pretty sure every cloud provider will support this just fine currently.
It fundamentally can't be improved in a significant enough way and still be Python. The more realistic options are to be okay with Python's type system (I personally am) or to look at different languages if you really want something more robust.
I agree with you, although I must say that as an occasional Python user the way type hints are implemented absolutely baffles me. In particular the fact that they are completely ignored by the standard implementation. They're glorified comments.
Given how opinionated the Python maintainers can be, it baffles me that they accepted to get these optional, noisy, half backed type hints into the core language.
In my experience given that they're optional and you'll almost never get 100% of your code and its dependencies with correct and up to date signatures it's just a nuisance. Maybe in the right projects if all the devs are very thorough with them it can be helpful, but that's really not my experience. If at least it triggered an assertion at runtime when the type doesn't match it would be massively more useful. And even then, if you're so thorough with your typing, why not just use a proper statically typed language?
I really don't get why it's even there to be honest.
Type hints are not perfect, but IMO after using them for a while, they are significantly better than nothing.
If you do the legwork, you can get mypy[0] type-checking done on your codebase in a way that is similar to how TypeScript does type-checking. There are stub files that are provided by project maintainers that are then used by mypy to infer things like e.g the return type of a function.
Type hints are also inline in the code, and not technically comments. They can be retrieved from class attributes using facilities from the standard library [1] and can facilitate other tooling that is specific for your project or that are more general.
> And even then, if you're so thorough with your typing, why not just use a proper statically typed language?
That would remove a lot of the benefit of choosing Python. Python is dynamically typed. Type hints make it possible to do type checking statically, but with a lot of extra leg work (as I described above). Making Python itself statically typed is not something that would interest 99% of the Python community IMO.
> the fact that they are completely ignored by the standard implementation. They're glorified comments.
Not quite. You can reflect on and interact with them at run-time, too. This does make it possible to implement run-time checking as a library solution.
I can't speak to why run-time checking wasn't built into the language as a default behavior, and the PEP doesn't explain why (though perhaps the answer is in the mailing list), but one possibility is that it would be wasteful. Most functions that are only prepared to accept certain types already have some form of run-time type check baked in, either explicitly or incidentally, so adding a second check would likely introduce overhead without producing a whole lot more type safety.
> I can't speak to why run-time checking wasn't built into the language as a default behavior
tldr: you other get either huge overhead or need to type annotate everything like it's Java, and it'll still be pretty limited/not super useful
---
there's a bunch of libraries that do this, and they're all fundamentally limited [though i guess some of these may just be failures of imagination on my part]:
- hard to check generics: if it's a builtin, you have to traverse the whole structure and check every element. and for user-defined collections, the programmer has to to manually instantiate every generic like:
CoolSequence[int]([1,2,3])
(which sucks) and every method call/property access needs to be checked
- if your function takes a callback, there's no way to typecheck that callback at runtime (without calling it, at which point it's not terribly useful. which extends to all functions i guess)
- not much you can do with iterators/generators. can't check if something really is an `Iterator[int]` without exhausting it, and e.g. type-erroring after processing 20 elements because it yielded not-an-int isn't very useful
so in the end, even if you wrap Every Damn Thing in a type assertion, you still get the errors very late, not much sooner than your usual "AttributeError: object None has no property named 'foobar'". kind of a worst-of-both-worlds situation
I'm pretty sure the level of type annotation you'd get roped into doing would be about on par with what you get in Java 10: function parameter and return types, but not local variable types.
My view is completely opposite to yours. I find type hints extremely useful and every new line of Python code I write uses them. I finally no longer have to do type checking / duck typing in my head but can simply write the types down. Moreover, an IDE like PyCharm will then do the type checking for me, notifying me of potential errors. (Even if I decide not to use Mypy.) Third, I finally only need to look at the signature of standard library functions to see what parameters they accept / what values they return. Previously, I always had to consult the Python documentation.
Bingo. I think a big explanation of the wide variance on opinion about type hints is due to reliance on tooling, or lack thereof.
Writing type hinted python in Pycharm or Ipython feels like racing in a sports car. Without it, feels like stop and go traffic with the occasional car accident.
I can see how if you are using a WYSIWYG editor, type hints feel like more text for little gain. But even then, it really helps document intent. In fact switching to IDEs is partly what made me go from someone who hated static typing to loving it.
> Writing type hinted python in Pycharm or Ipython feels like racing in a sports car. Without it, feels like stop and go traffic with the occasional car accident.
This is a beautiful analogy!
As for the second part, though, I would say that even if your IDE does not provide automatic type checking, type hints will still add some benefit because:
- without type hints you have to do all type checking in your head (across all stack frames).
- whereas with type hints you only need to do type checking down to the first level of the stack.
To illustrate what I mean, consider the following code:
def bar(param1, param2):
... # Some complicated code from which it is not immediately clear what values are allowed for param1 and param2, nor what the return type of the function is.
def foo(param):
return bar(param, param2="some default value")
foo("something something")
Now, in order to verify that `foo("something something")` is correct, you have to manually look at foo() and then at bar(). Needless to say, this adds a lot of mental overhead to the game. If we added type hints to both foo() and bar(), however, you would only have to check the signature foo() – which, even in IDEs without type checking, usually amounts to hovering over `foo("something something")`.
> Given how opinionated the Python maintainers can be, it baffles me that they accepted to get these optional, noisy, half backed type hints into the core language
Guido joined the Mypy team while he was Python’s BDFL.
Also, there’s nothing half baked about either Python’s type hints or Python’s type system, or it's major type checkers. It's not Haskell, sure, but it's an expressive to system, the typecheckers are reasonably smart, and the annotations are readable and sensible if somewhat verbose; there were some infelicities regarding alternate names for core types in annotations, but that's been improved recently.
> In my experience given that they're optional and you'll almost never get 100% of your code and its dependencies with correct and up to date signatures it's just a nuisance
In my experience they start to provide value in preventing bugs and easing development because of tooling support way below 100% coverage.
> If at least it triggered an assertion at runtime when the type doesn't match it would be massively more useful.
Python’s type annotations are annotations, and are used by some libraries for runtime (validation, serialization/deserialization, etc
) as well as static checking purposes (e.g., pydantic.)
> And even then, if you're so thorough with your typing, why not just use a proper statically typed language?
All a “proper statically typed” language is is a language with a static type checker run ahead of time, which Python is of you choose it to be. There's a lot of code in the ecosystem that is more broadly types than it needs to be, because no annotated code which checkers can't infer anything better for use Any, but that's evolving over time as it is more common for popular libraries to be typed, or at least have typings available.
The chorus of people asking for increased typesafety had become too loud to ignore. Type hints gave them enough support to cover 95% of their needs, which were/are largely organizational rather than technical: developers must feel sure they are using the right classes in their code, pulled from the right places in their project, and don’t have to look up docs at every step (because the IDE will autocomplete stuff). What happens later, the low-level implementation, it doesn’t really matter; what matters is the information about types is surfaced somewhere, so that IDEs and tooling can use it to document projects and help developers.
That’s what type hints do, and yeah, they are basically glorified docs, but integrating docs into syntax is one of Python’s many traditional strengths (see docstrings, doctests, etc). That’s also why they are optional, thank goodness, so people who don’t have big-org / big-project needs can still be productive.
This was always kinda false (type hints can be composed unlike eg docstring param @type). But with the introduction of dataclass, this is empirically false. You can readily hook into the type hints, reflect on them, use it to marshal data, etc. Using __post_init__, you can leverage any kind of runtime checks you want to enforce invariants.
Also type hints greatly help transpilation.
You don't need 100% to reap the benefits. In fact I will annotate small bits of un-typed code and it makes it way easier to reason about.
I don't think that's accurate. Python's type system is largely defined by mypy + the annotations, in addition to runtime python. In that regard, python's type system has been evolving really quickly, and it's becoming something quite interesting. I'm a fanboy of static type systems, and in the last bunch of years, python's type system has gone from just bearable to really useful. I'm excited to see where it goes from here.
The dependence on library authors is always a challenge in any language. You might have one author using `[a]` where another uses `PositiveNumeric a, Fin n => NonEmptyList n a` for the same thing. You can always just annotate whatever the library author used (e.g. they return a list of strings, so you use List[str]).
There are some interesting further add ons that seem very python, allowing you to go further. For example, with a pandas dataframe you can just say your type is a dataframe which isn't so useful, but it's possible to hack your own types onto it in the vein of https://github.com/CedricFR/dataenforce, or use things like https://smarie.github.io/python-vtypes/ to get smarter typing on things the authors didn't type. I expect that trend will continue.
What fascinates me about python's types is actually the very fact that they are bolted on. You have a language that lets you do crazy things and a type system trying to catch up and make it convenient to verify those crazy things. It's a nice complement to the usual developments of verifying all of the things and slowly extending the set of things you can do.
> You might have one author using `[a]` where another uses `PositiveNumeric a, Fin n => NonEmptyList n a` for the same thing. You can always just annotate whatever the library author used (e.g. they return a list of strings, so you use List[str]).
The difference between these two cases is that a function that returns [a] will always return [a] and if you convert it to `NonEmptyList n a` you will be nudged towards handling the case where it didn't actually return n elements, whereas if you annotate someone else's list as returning List[str] then it may well silently not do some of the time, and you won't get an error until a long way away.
I was unclear. I meant the other way around. In the case that the third party code does in fact always return no empty lists of positive numbers, the author might just type it as a list. A third party author in even a language like Haskell won’t necessarily use the bells and whistles of giving very specific types.
And by the List[str] bit, I mean you can annotate it as doing whatever it in fact does. You aren’t protected if you get that wrong, of course, and that’s where things like stubs come in to help.
> In the case that the third party code does in fact always return no empty lists of positive numbers, the author might just type it as a list.
Yes - but as I said, you can be confident that the list type is accurate, and if you want to convert that list to a specific-length list type then you're nudged towards doing it in a way where you handle the case where the list isn't actually the length you specified.
> And by the List[str] bit, I mean you can annotate it as doing whatever it in fact does. You aren’t protected if you get that wrong, of course
Not "of course"; the whole point of a type system is that it's automatically checked and therefore you can have reasonable confidence in the types you're given.
They don't really though? Stubs are just as unchecked as writing the type signature yourself, so even if they're correct at a given point in time, they'll tend to become incorrect for subsequent releases of the libraries they cover.
For what it’s worth, TypeScript had somewhat of the same problem, and yet these days it’s rare for me to find a well known library that doesn’t include type definitions!
> It fundamentally can't be improved in a significant enough way and still be Python.
Why not?
Static enforcement of invariants that doesn't change the space of what is possible in the language when you aren't asserting conflicting invariants doesn't make Python any less Python.
GP was referring to type hints and how they don't provide any runtime validation. There are libraries like Pydantic that let you do some, but nothing built into the language runtime IIRC.
First, managing python dependencies and versions, current system is horrible. Focus on fixing that, then performance. Swift is much faster than Python. Third, Swift is a much advance and intelligent language to program in than Python. When I write Swift code, I feel like a PhD and when I write Python, I'm back in elementary school.
It doesn't matter that "Swift is faster than Python". Your TensorFlow graph is gonna be slow as all hell if it's running on the CPU. The bottleneck here really isn't performance on a CPU. That's a no-brainer for all non-ML applications.
But for applications that require a GPU (i.e, most ML applications) cutting over to Swift from Python will likely win you nothing in performance, and wouldn't be worth it at all.
So one of the big ones is dynamic method replacement. In Swift it's possible to declare a method `dynamic`, which means it will always be dispatched dynamically. It's then possible to load a dylib which "overwrites" the method at runtime, making it possible to implement hot code reloading in an elegant way which is fully type-checked and everything. I am not aware of a way to do this in rust.
Most of the other things I miss are ergonomic:
- Swift has better inference, so for example in a match statement over an enum, in rust you always have to type: `MyLongEnumName::SomeEnumCase`, where in swift you can just type `.someEnumCase`
- Swift's operators for optionals are much cleaner imo than how it's handled in Rust. For instance, the `?` operator is kind of magical in Rust, because it effectively changes the flow of control of the enclosing function. Optional handling constructs in Swift are always local to the statement, which I find to be easier to reason about, and more composable.
- Rust's module system is needlessly complex and verbose, and adds boilerplate and redundant busy work when you want to refactor code and move things around
- Trailing closure syntax is really nice
- It's sometimes useful to have types as runtime constructs as well as at compile time
- I like Swift's more flexible approach to scopes/namespaces. Like in Swift, when I declare a struct, I can just declare the methods inside the struct body without needing a separate impl, which reduces boilerplate. Also for instance if you need to declare a throwaway data type which is only used internally by a couple functions in a Swift type, you can declare it right inside the struct declaration where it's relevant. Rust is more restrictive, and I have to declare types either at the module level, or inside an execution scope like a function body. So with Swift I just feel like my code is organized in a more semantically coherent way which reads like a book, where in Rust it's more organized according to the ceremony which is required by Rust syntax.
I could go on, but basically I just feel that when I am coding with Swift, the syntax melts away and I am mostly focused on the problem domain. When coding in Rust, the syntax is very present, and is a big part of what I am dealing with.
Back when I was doing a lot of Swift programming, this seemed like such a great idea. Since then I moved on to Julia. And when I looked at the ML code for Swift and compared it to Julia, I was thinking "How on earth could anyone think this was ever a good idea?"
Doing machine learning stuff in Julia is simply much more user friendly than doing the same in Swift. Swift is nice for iOS development, but I think in data science and Machine Learning Julia will always be a much better choice.
It is a pity Google did not go for Julia. Julia has gotten exceptionally far despite limited resources. With Google style resources, Julia could have been massive.
Despite modest investment, the Julia JIT beats almost anything out there. It would have been amazing to see what they could have pulled off if they had put the kind of resources that was put into the JavaScript V8 JIT into the Julia JIT.
Julia uses a method JIT, which are quick to implement, but which makes latency gets bad. Meaning if you load a whole new package, running the first function can cause some delay. With a JavaScript style tracer JIT added into the mix one could have probably managed to have both high performance and low latency.
Alternatively with more investment they could have had better precompile caching, meaning libraries that had been loaded in the past would JIT really fast. Julia does that today, but it could have worked a lot better than it currently does. It isn't that it can't be done, but it just requires resources.
IMO, Julia is good enough that you might not notice. you can use constructions like `for i in axes(V)` that will automatically give you the indexes, or `V[begin:begin+2]` to get the first 2 elements. Also higher order functions eg. `map` (and broadcasting) mean that you will rarely be manually working with indexes.
That's not so obvious, the abstractions for multi-dimensional indexing are better than (I think) you're imagining.
Here's a nice recent example: a new reverse!(A; dims) function, which is fast and works on arrays with any number of dimensions. And is completely indifferent to where the indices start:
Maybe less readable than I remembered, sorry! But these `CartesianIndex` things are how you index multi-dimensional arrays.
If by indexing you only mean adding offsets to pointers with your bare hands, this is possible but discouraged. Because you'll produce a buggier and less general version of exactly what already exists. In this case, reversing an N-dimensional array, along any M of its axes, with arbitrary offsets -- there are a lot of ways to get that wrong.
# one dimensional
a = [1, 2, 3]
# prints the number 1 @ index 0
print(a[0])
# multi-dimensional
b = [[1, 2, 3], [4, 5, 6]]
# prints the number 4 @ index (1, 0)
print(b[1][0])
OK, I thought you were advocating zero based on complicated index calculations for some algorithm, with mod & div, which is what I was trying to say could be abstracted away. The literal syntax `a=[1,2,3]` makes a one-dimensional `Array{Int,1}` whose first element is indeed `a[1]`. (`b` is an array of arrays, while `c = [1 2 3; 4 5 6]` is an honest 2-dimensional array.)
Because you believe that you can't get used to a different convention? It gets brought up a lot that for array offsets, zero based is more natural. In my experience, the amount of index calculations you need in Julia is minimal anyway because there are really nice abstractions for n-dimensional indexing.
0-based indexing is an artefact of pointer math. Modern languages don't treat arrays as a pointers to their first element. So, it is still with us only because almost no language designer decided to challenge existing convention.
I see 1-indexing as an artifact of a lot of linear algebra texts, whereas 0-indexing is more popular in analysis. Julia just wants it to be easier to directly copy formulas from publications.
I also think it’s blurry as to whether mathematicians and programmers see indexing numbers as merely indices or richer numbers.
The beautiful thing of having polyglot experience, since the days when C was just another language trying to get users, is not being put down by minutia like that.
> Julia uses a method JIT, which are quick to implement, but which makes latency gets bad. Meaning if you load a whole new package, running the first function can cause some delay. With a JavaScript style tracer JIT added into the mix one could have probably managed to have both high performance and low latency.
This is actually the other way around -- Method JITs are harder to implement and all the major JavaScript JITs are method JITs, not tracing JITs. There are many reasons that contribute to the latency but I don't see what makes method JITs inherently slower than tracing JITs.
I like Julia a lot, and I think it's the best option for ML research, but isn't there a big advantage to Swift or similar languages when it comes to production deployments on mobiles? The distribution of compiled binaries in Julia seems a bit kludgy and less reliable than with a statically compiled language like Swift.
S4TF lost out, not to Python, but to Python's AI/ML ecosystem -- people, projects, libraries, frameworks.
Despite its many shortcomings, Python has become the lingua franca of AI and ML, to the point that whenever I come across a newly published AI or ML paper that I find interesting, I expect to be able to find code implementing it in Python.
For example, yesterday I saw a post on HN about approximating self-attention matrices in transformers, which have O(n²) computational cost, with a seemingly clever approach that has O(n) computational cost: https://news.ycombinator.com/item?id=26105455 . "Huh, that looks interesting," I thought, "let me see if I can find some code." Three clicks later, I found myself at https://github.com/mlpen/Nystromformer -- the official implementation... in Python. A few moments later, I was playing with this thingamajiggy. No other language comes close to having this kind of ecosystem in AI and ML.
Julia still has a shot at becoming a viable alternative to Python, especially if the Julia developers can shorten the "time to first interactive plot" to make it as fast as Python's, but they face an uphill battle against such an entrenched ecosystem.
This is a false equivalence of the first order. What makes JS impossible to dethrone is that whatever replaces it would need to be supported by a critical mass of web browsers, which just won't happen.
AI/ML on the other hand has no such restriction. Your new startup could build all their ML models in Java with DeepLearning4J and it would probably work fine. Python is used because it's easy to work with for experimenting, which is a big part of research and for production your can extract your weight and graph structure to run on TVM/TensorRT/ONNX. There is no equivalent vendor lock-in.
Don't forget Web Assembly. That could easily dethrone JS but it will take time. All the major browsers have been evergreen for some time now so it's not like we're back in the early 2000s.
Well you could also build your web startup in a compile-to-JS language and barely ever touch JS, and you’d also be fine, right? (Maybe in rare cases for interop, debug, etc)
At any rate you’re right - the dominance of each language for their respective use cases (JS-Web / Py-ML) emanates from different sources.
Sure, but isn't it nice that you don't even have to read the paper to play with the full model, to see if the claims hold up, using the same Python tools you and everyone else uses?
To be honest that is not really true for many ML implementations from papers. Often, I will pull a repo and it will be half-baked, or, only works on python 2.4, or have a broken dependency on something or another (zmq was common issue for a hot sec), or not publish weights, or be in tf when I want torch, or be in torch when I want tf, etc.
Yes, I agree: a lot of code is indeed half-baked! But good papers tend to have better-than-average code, and ~100% of the time, the code is in Python, not some other language.
Swift for TensorFlow was a nice idea on paper, but you need more than ideas to push a project forward. You need people who actually want to use your product.
For all the comments on HN about how great Swift was, there were hundreds if not thousands of ML engineer and research scientist who did not know it existed and frankly did not really care about it either.
Swift for Tensorflow was not addressing the issues of ML in productions and felt like "Just another framework" but with the added handicap of needing its users to learn an entirely new language which is not something most scientists want to do. One might argue that engineering might be more interested, but in a lot of cases the science team will push the models and then the engineering team will put them in production, leaving very little room to migrate from TF/Pytorch to Swift for TF.
It's interesting to compare Swift to Julia here – Julia had a base of people who absolutely loved it from the 0.2 days, and has managed to draw many users from Python/Matlab/R. There are many reasons for that, but overall Julia had something that really appealed to scientists: syntax they liked + competitive-with-C performance.
In contrast, S4TF's target audience seems to have been Swift developers, and they didn't really appeal to the data science crowd.
Precisely, S4TF felt like a startup product with no market research. It really felt like the only reason Swift was chosen was because the main dev created it.
We see this all the time right, the next big ML company is going to come from some scrappy startup with a precise focus on what to do (either because they're geniuses or because they're just the lucky idiot amongst 10,000 ML startups), not from some behemoth that decides "we'll eat that for breakfast". Swift4ML suffers from the fact that Swift exists. To make S4ML work you need to abandon atleast some of the things that makes Swift Swift.
Swift now being a 100% Apple-sponsored & owned project again makes me a bit nervous.
Anyone knows if chris latner is at least using swift in his new company ? I have the feeling swift never really worked in the server side, data science is now officially a failure, and all that is left is now a very niche market of 100% native mobile development.
I love this language, but i'm eager to see it handled by a proper foundation with big names behind, as it's a really great language.
>a very niche market of 100% native mobile development
I mean, it also works for native desktop development. And is there really an issue with that? Objective-C basically had no reason to exist beyond iOS/Mac programming and at least now we don't have god damn [myString stringByAppendingString:@"another string"].
I suspect that we'll be seeing ObjC for a long time, as the lower-level system language for Apple devices. I know that Apple still uses it for much of their system programming. No idea if that's by choice, or legacy (possibly both).
I was looking at the upcoming async/await stuff for Swift, and it's still not comparable to some of the lower-level threading systems.
That said, I have not done system programming for a long time (unless you count drivers). I think Swift is an almost ideal application-level language.
I'm not especially bothered by it being Apple-only. It works on all Apple-supported hardware, has a new application toolkit coming into play (SwiftUI), and has a very active development community. Spend some time on the Swift discussion forum to see some pretty heady discussions.
I'm a little bit confused by this comment. Obj-C is, in a lot of ways, the _higher_ level language on Apple systems now, and most of what I would consider "systems" programming on macOS and iOS is in C, assembly, and C++.
You can more or less write C in Swift as well, though. Some pointer operations get more verbose, but the standard library also makes some stuff easier. The only thing that's really missing is a nice fixed-size array syntax (they get imported from C as tuples, which is not ideal).
I suspect we will hit an inflection point where Objc is bridged to swift code rather than the other way around - no reason to let Swift be the one taking the interoperability hit.
There are already controls in Big Sur and iOS 14 (Color Picker) which are SwiftUI and bridged to UIKit/AppKit.
We'll see it in legacy code for a long time, sure, but there's every indication that all new Apple frameworks are being written in Swift. Anyone starting a new project in Objective-C is in the minority.
If you look closely at how that data was collected, what it's actually measuring is whether a binary links against the ObjC runtime library, which will be the case if a binary transitively depends on any ObjC framework, so even if all new frameworks and executables were written in Swift, we would still expect to see the number presented in that post to continue to grow until most or all of the important frameworks using ObjC are rewritten entirely in Swift. I don't think this data is sufficient to say one way or the other to what degree that is occurring.
Fair criticism. I'm not sure why the author didn't try to detect the use of objc_msg functions, for example. So the ObjC binaries may be overcounted a bit.
Still, the test for Swift binaries seems accurate, and if you look at iOS 13.1 vs 14.0, for example, according to the chart there was an increase of 157 Swift binaries and 446 Objective-C binaries. If we assume there are 157 "false positives" in the ObjC binaries, that's still an increase of 289 ObjC binaries that don't use Swift at all.
D'oh, right. A good test might be difficult. In the absence of that, I guess the safe bet is just to count every Swift binary as a false positive for ObjC, though that's not quite fair, since you can mix the two in the same code base.
The number of binaries of all languages in iOS (except C) increases year over year, so that doesn't really mean anything. It's more telling that the number of Swift binaries in proportion to the number of Objective-C binaries is increasing year over year.
We'll see. Off the top of my head, only Combine and SwiftUI are Swift only for now. And Combine has specific features that are explicitly meant to make it interop with ObjC (and @objc Swift) objects, e.g. KeyValueObservingPublisher. The other "hot new" Apple frameworks, like CoreML, ARKit, HealthKit, ResearchKit, etc. are all still written in and usable from Objective-C. They _are_ starting to use Swift(UI) in UIKit, but so far continue to wrap it in an Objective-C interface.
not really.. games are made with 3D tools (unity, etc), Form-based "data oriented" apps are made with cross-plateform tools (react native, xamarin, flutter, etc).
The only things left are a very small percentage, and add to that the fact that you either 1/ have the budget for 2 development teams, or 2/ afford to skip the other "half" of the market staying iOS only.
> I have the feeling swift never really worked in the server side
Vapor [1] is wonderful to work with and has a large user base. Apple is also putting significant resources into Server Side Swift with Swift NIO [2]. Lots of really cool stuff happening in that ecosystem
I don’t think swift’s performance is good enough to really make a dent in that area vs rust/java and C. Nor are its macro-programming ability enough to compete with dynamic (and slow) languages.
Now there’s still go market of « server middleware ». But for that it would need a much higher quality of core libraries, tooling, and a much better concurrency story. Maybe it’ll come with the incoming actors, but somehow i doubt it (the underlying concurrency libraries remaing grand central dispatch, which looks too heavy compared to, for ex, goroutines)
Swift lovers have to grasp that unless Apple actually puts the effort like Microsoft started doing with .NET, it is going to be used outside Apple eco-system as much as Objective-C has been used in the last 30 years.
It seemed like Swift looked like a promising language for data science a few years ago. I'm not familiar with Swift. Can anyone provide additional context as to why the language seemed promising for data?
I disagree with the pitch, but the pitch I heard when this was announced was:
* Swift lets you write code that's as concise as Python while still being fast, and therefore allows you to get away from the problem of having to write ML code in multiple languages (TensorFlow is actually mostly C++, which makes it difficult for a Python developer to debug.)
* That the lack of static typing is a major pain point in Python, and Swift's type system would help write code with fewer errors.
> Swift lets you write code that's as concise as Python while still being fast,
> (TensorFlow is actually mostly C++, which makes it difficult for a Python developer to debug.)
To be honest, if I had to choose between Python and Swift I'd still choose Python. Swift is a nice evolution from Obj-C and all but it is nowhere near as simple as Python, or Ruby, or PHP, or Javascript, or Kotlin. And Apple's documentation of these things is pretty woeful at the best of times lately.
I also disagree with the pitch, is what I'm saying.
To say that Swift is "nowhere near as simple" as Javascript or Kotlin is a very strange opinion to me, and I have experience with all 3.
Simple as in syntax, memory management, or in what regard? I can't think of a single example where Swift is not equivalent or better. Do you have the same criticism of Scala?
Swift has no traced garbage collector which means you cannot represent circular data structure/references.
Also swift support for interfaces (protocol) is sub-par with today standards
No support for default implementation methods.
No support for generics in protocol.
There might be workarounds but for a newcomer, this is a red flag of reinventing a basic and universal construct (interfaces) in a sub-par one, this show a dangerous mentality from the swift vm devs, what else did they wrongfully reinvent in a sub-par way?
I think considering swift as merely an evolution of obj-c is a drastic underestimation. Swift is a very nice language in its own right. Having spent time with most of the languages you've mentioned, while the learning curve might be a bit steeper, I think Swift is much easier to manage a medium or large project with than any of those options with the exception of maybe Kotlin, and personally I find some of the design choices a bit better in Swift than in Kotlin.
I don't get the hype about Swift. There are at least 4 features in Swift I can think of that are just plain clunky:
1. Backslashed opening parens for string templating
2. Parameter labels and the use of _ when ignored
3. Splitting method names across opening parens eg. `move (to ....`
4. Verbose NSString Objective-C hangover in regular expressions
Those are really minor syntax details (except maybe the regex stuff, that’s more a library issue than a language one).
Swift is safe (no null pointer exception), has predictable garbage collection performance through ARC, has a familiar C-style syntax (easier to get on-board), has compile-time type checking, good-enough generic programming support, algebraic data types with pattern matching, and will very soon get actor support.
Plus, it has the potential to become a good cross-platform language since it’s using llvm, all it needs is a bit more love on tooling and libraries.
I don’t see many other languages with those characteristics..
Swift just seemed like yet another OCaml-like language to me; there are certainly far worse languages to borrow from, but I don't see any compelling reason why I'd use it over OCaml. ARC has the same worst cases as mark/sweep GC AIUI; exiting any scope might cause an arbitrarily large amount of cleanup work in the general case, and it doesn't solve the memory fragmentation problem which is the main reason you still need to occasionally stop the world (ish) in a mark/sweep GC.
I agree from a technical POV, but there is a social reason (network effect): Swift has Apple-backing, OCaml is not currently supported by any of the big,
influential software companies. OCaml's main backer now is Jane Street
who do a lot, but is too small, and Facebook's support (via ReasonML)
is too half-hearted to be compelling. As a former OCaml programmer, I
would not currently bank a career or startup on OCaml. Indeed I've
been doing OCaml programming over the last few weeks (and seen the shortcomings of OCaml's current library eco-system vis-a-vis more mainstream languages) in order to
interface with a big existing OCaml development, and they told me they
decided to move to Rust for all future new developments.
I say this with a heavy heart, as somebody who spent his first decade
as a programmer with OCaml.
Interestingly, even in Feb 2021, there is no compelling mainstream alternative to OCaml (in the sense that e.g. Jane Street are using OCaml):
Haskell's being lazy, Java/Scala etc as well as F#/C# being JIT'ed
make performance predictability difficult. Rust does not have this
problem, but for many tasks where software engineering agility is a more
important consideration that extreme performance (which might be most in-house business software?), a GC'ed language
would appear to be more suitable.
Thanks. I was not aware of this, since I've not worked on a Microsoft stack for a long time. Are those F# AOT compilers mature and support all features, including libraries? In this case F# would be a viable alternative to OCaml for Jane-Street-like tasks.
yeah, they're doing the world a favour by taking the code and conform to its licence, as opposed to... developing a new browser engine which is... easier?
WebKit got forked into Blink, which is no longer under Apple leadership. Brave, Chromium, and the other major KHTML-derived browsers adopted this.
I can think of dozens of examples from Microsoft and Google, which are more "outside ecosystem" inclusive as a means of gaining mindshare. TypeScript, Golang, Visual Studio Code, protobuf, gRPC, ...
Other companies think of the things living outside their moat more flexibly, or their moats are less rigid. That's not to say that Apple's approach is wrong (look at their market cap!), but it has different consequences in terms of the open source code mindshare they cultivate.
Apple churns languages every few years? They kept Objective-C around for 13 years before introducing Swift, and that was a language that was widely disliked by people from outside the Apple ecosystem. You can still use it for the vast majority of OS features today[0], nearly 20 years after it was first used on Apple platforms. Heck, Obj-C is used for Mac/iOS development because that was the preferred language for Mac OS X's immediate ancestor, NextSTEP, which means that it's been a part of the same development ecosystem for about 30 years. If this is churn, I think I'm okay with it.
[0] As far as I know, the only system feature that requires Swift is iOS 14/macOS 11's new widgets, which must be written in SwiftUI.
Next is not MacOS so I think it’s a real stretch to claim that means long term support! Perhaps some of this is normal but I think Apple have made little effort to avoid churn and it suits them to keep things changing.
Apple have had significant churn in dev tooling, languages, sdks, frameworks, chip architectures over the last couple of decades. Perhaps that is completely normal but it does mean developers are always struggling to keep up, and an app written for iOS 1.0 needs significant alterations if not a rewrite to work with later versions. Almost everything has been completely replaced, from language to frameworks to dev tooling like resources.
Right... Because Apple has done that so many times... oh wait they have done it exactly once since re-launching after acquiring NeXT. IT was Obj-C and now it's Swift except they actively support Obj-C still, so I'm not sure what you are talking about?
Java was introduced alongside Objective-C, with a common bridge to call into Objective-C runtime, as Apple was unsure if the Apple developer community would be welcoming to Objective-C.
They dropped it after the community was more than happy to adopt Objective-C.
About the same time they introduced PyObjC support and for a while there was MacRuby, which due to internal politics was eventually dropped, the creator left and now sells RubyMotion for mobile development, originally based on it.
Then if we switch back to the System days, there was Object Pascal, replaced by C++ frameworks, Hypercard, Lisp, Dylan.
Nice shifting of the goal posts. Introducing a possible technology while supporting your main one is hardly the same thing. Since Next became OS X Objective-C has been supported and it has been the main focus of development until Swift. Apple is a different company after Next, the System days have no real bearing on modern Apple. Java, PyObjC, and MacRuby were all attempts to offer more ways into the main eco-system. They were never attempts to support a new dev environment. Swift is the only time Apple has done that since Jobs returned.
> Swift now being a 100% Apple-sponsored & owned project again makes me a bit nervous.
Are you nervous about a language that is designed and supported by the largest and most successful company in the world?
> very niche market of 100% native mobile development
That "niche" is at least 1.5 billion devices. Devices bought by the 1.5 billion richest people in the world because they like them, not because their employers forced them to. Seems like a pretty ideal "niche".
But I wouldn't agree that "swift never worked server side". I've built several backends in Vapor and it's amazing.
> Are you nervous about a language that is designed and supported by the largest and most successful company in the world?
Yes, I am. The concentration of power between facebook and google for ML frameworks is worrying enough, but to their credit, they have so far been very open and collaborative and are giving a lot back to the ML community in terms of research
Apple on the other hand is the dictionary definition of walled garden. Their controlling and insular nature has been the reason for their success, and power to them, but for something as fast moving and community driven as machine learning, I would be very wary if relying on them for any part of it.
I think there is too much focus on Swift as a general purpose language. I really enjoy working in it and it’s great at what it does, does it have to be deployed in every domain?
What possible future are you trying to ward off here? Swift is an open-source project, it's self-evidently in Apple's interest for as many people as possible to use it.
Be concrete, please. What scares you about using an open-source language which is mostly developed by Apple? Do you have the same paranoia about clang? If not, why not, what's the difference?
iOS doesn't come as 100mb cellular downloads, so that isn't really the same kind of issue. There are many reports about showing how it's slowly coming into wider spread use at Apple.
For me the lesson is that just because a new technology/language is out doesn't mean you should jump on it. Things need time to mature, and if you're Uber, you can't risk having half of your income rely on something new. Compilers, linkers, and programming languages (and databases and OS kernels) take years to mature. Hell, just earlier this week was a post about diagnosing a 21 year old bug in the Linux networking stack for rsync of all things.
I'm quite shocked that enough experienced people felt that level of confidence. In earlier versions of Swift I personally experience slow compiles and that was on a not terribly large code base -- no where near Uber. That alone should have been a big clue of the state of things.
Uber doesn't have any pressure to have a reasonable tech stack because they don't have to be profitable (because they're an investor charity). On the other hand, having a fancy tech stack helps with recruiting.
Reading that thread, I'm always so confused by HN posters who ask for evidence that would be impossible to provide without self-incriminating. Are they asking disingenuously, or are they just clueless?
Always makes me laugh. Perhaps something like 10% clueless, 80% disingenuous and 10% feel they are somehow entitled to information that's obviously sensitive or private.
No surprises there. There were maybe a total of 5 people excited about adding automatic differentiation to Swift. Too bad they didn't try improving Julia instead.
I remember their terrible article trying to justify why Swift was the right choice out of a load of more appropriate languages. I'm still annoyed they didn't go with Julia.
I always figured they would have chosen rust if it weren't for Lattner's sway (real or perceived) in the Swift compiler development community. One underestimated advantage of rust over swift in my view is that it has interested practitioners across a much more diverse range of problems spaces. I think it was always going to be hard to convince people to use a language for for ML when most of its practitioners are solely mobile developers. But people already used rust for lots of different kinds of data processing heavy use cases.
You can't be serious. Rust would just replace C++ in the current formula, you can't use Rust as if it was Python. ML people don't need to waste time proving memory properties at compile time, cmon!
Indeed, there were alternatives. Two already come to my mind, one is Scala (which syntax-wise is strikingly similar to Python) and Julia. Both having features like type-safety, macros, JIT, FFIs, etc.
None of this is less true for swift. The whole idea is that there may be an advantage to better static analysis that overcomes the perceived downside of using a language with a strict compiler and lots of rules. That idea may or may not be worthwhile, but it is equally applicable to swift and rust.
Julia I agree but Scala strikes me as a bit of a non sequitur. It is much more complex and foot-gunny (and thus difficult for scientists who only know python to get right) than either swift or rust, and much more difficult for static analyzers.
Scala already has a foothold on the same research domain due to large data frameworks like Hadoop and Spark, with NVidia caring for polyglot support on CUDA, including JVM based languages.
That is true, and certainly relevant, but I believe it is also true that Scala does not meet the criteria for what that were trying to do, because it is not easy (perhaps not possible) to do the kind of static analysis necessary to achieve their goals.
For a generic "what language with a good type system should we use with ML?" project, I would totally agree with you that Scala is an obvious choice. But this project was attempting a more specific thing, for which Scala is a less good fit.
Neat! This may have not been well known when they kicked off the project and wrote their reasoning. Here is what they had to say about Scala at the time of the document linked up-thread[0]:
"Java / C# / Scala (and other OOP languages with pervasive dynamic dispatch): These languages share most of the static analysis problems as Python: their primary abstraction features (classes and interfaces) are built on highly dynamic constructs, which means that static analysis of Tensor operations depends on "best effort" techniques like alias analysis and class hierarchy analysis. Further, because they are pervasively reference-based, it is difficult to reliably disambiguate pointer aliases."
If they were wrong about that, or if the state of the art has progressed in the meantime, that's great! You may well be right that Scala would be a good / the best choice if they started the project today.
While I love using Julia for many things, I prefer having a Python monoculture for AI to be able to mix and match different algorithms.
At the end the engine that compiles the mathematical expression to hardware is what matters, and I don't think that LLVM IR that uses is the best IR for the optimizations.
> I prefer having a Python monoculture for AI to be able to mix and match different algorithms.
I do not. Unfortunately, high performance and python do not go hand in hand. Yes, I know the heavy lifting is done by C/C++/Rust/Cuda/Blas/Numba and so on, but, when you run simulations for millions of steps, you end up with billions of python function calls.
Afaik only Jax actually performs any optimizations because it constructs an analytical gradient. Zygote seems to be able to that and more on the LLVM IR level which, I think should enable more optimizations.
I'm not sure I understand what your getting at. If Python's performance in the ML space is not sufficient, then the community would have quickly moved on from it and built something better.
And that something better is certainly not Julia and it's definitely not Swift.
> Yes, I know the heavy lifting is done by C/C++/Rust/Cuda/Blas/Numba and so on, but, when you run simulations for millions of steps, you end up with billions of python function calls.
For ML purposes, if your model isn't running on the GPU, it doesn't matter if you're using Swift, Rust, or whatever, your stuff is gonna be slow. Like it or not, Python is one of the best glue languages out there, and the reason why libraries like TensorFlow and Torch are not used in their native languages (C++) [0] is because they're significantly simpler to use in Python and the performance overhead is usually one function call (e.g run_inference(...)) and not billions.
If you find yourself writing a simulation, and you need to optimize away billions of function calls, you can use the C++ API provided by TensorFlow.
>If you find yourself writing a simulation, and you need to optimize away billions of function calls, you can use the C++ API provided by TensorFlow.
I prefer to just use Julia and get that for free. Also can write custom cuda kernels in pure Julia. And differentiate through arbitrary julia code, compose libraries that know nothing about each other etc
Sure, that's fine. But I'm assuming GGP is talking about TensorFlow, since that's what this post is about. If you don't need TensorFlow, then this whole conversation is kinda moot, and you can use whatever you like.
So nobody needs TensorFlow, yet it and PyTorch are pretty much the only frameworks widely adopted for ML, and they're in Python. A quick Google search will tell you that, I don't really feel like rebutting to a baseless statement.
> If Python's performance in the ML space is not sufficient, then the community would have quickly moved on from it and built something better.
Python's performance is sufficient when the bottleneck is actually the computation done in the accelerator. In my flavour of ML, we use small models, think 3 layer NN 64 neuron wide and in some cases a small CNN. During training, most of the models reported using <150MB.
Most of the community finds python sufficient because they do not need to interleave training and simulating.
> For ML purposes, if your model isn't running on the GPU, it doesn't matter if you're using Swift, Rust, or whatever, your stuff is gonna be slow. Like it or not, Python is one of the best glue languages out there, and the reason why libraries like TensorFlow and Torch are not used in their native languages (C++) [0] is because they're significantly simpler to use in Python and the performance overhead is usually one function call (e.g run_inference(...)) and not billions.
You don't know in what I am running/doing, hence your comment comes off as ignorant.
This is the setup:
Run X number of simulated steps on CPU, collect X samples, store them in a buffer of size either X, or Z>>X, train for Y number of steps sampling from buffer, copy model to CPU, repeat for total of 1M steps, repeat 10+ times to get a good average of the performance.
All of that without any hyper parameter tuning.
Now, you also need to note that a non trivial amount of work is also done in python to augment the inputs as necessary. If the work is done in numpy, there's usually little overhead, but, it is often the case that the environment I am simulating is wrapped in wrapper functions that modify the behavior, e.g. I may need to remember the last 4 instances created by the environment, and so on. All these modifications quickly accumulate and for a single experiment of 1M steps, you end up with billions of python calls. The community uses more or less the same package/framework as the intermediary between simulations and models.
The issue is so prominent, that the community as a whole moved away from running in single core the simulations, to having multiple parallel actors to collect transitions/data points, this also requires new theory. Furthermore, there have been many proposed architectures for distributed and asynchronous training because the bottleneck is not the GPU or the model, but rather, how fast you can collect transitions. Infact, there was a distributed architecture by google that literally sends the transitions over the network into a few GPUs, the reason is that the network cost is amortized because you get to run hundreds of simulations concurrently.
IIRC, a maintainer of a popular project/framework that I contribute saw improvements upwards of 2x when using C++ over python.
> Most of the community finds python sufficient because they do not need to interleave training and simulating.
That's kind of my point. Most of the community has models running on GPU's and don't care too much about CPU-bound workloads, training or otherwise.
If you do care about that, you are in a relatively small niche of machine learning, statistically speaking. I am not denying it's existence, I'm just saying that your stack will have to be different if you want to extract the maximum level of performance out of your CPU.
> IIRC, a maintainer of a popular project/framework that I contribute saw improvements upwards of 2x when using C++ over python.
That's not surprising at all. Like I mentioned in the parent, if you profiled your code and you found that Python function calls are the main bottleneck, and you believe it to be worth investing time in getting rid of them, you can use the C++ API's of Caffe/TF/PyTorch/Whatever.
I personally don't work in simulations so I haven't ran into your problem. In the deep learning world, the CPU is unusable for any task (training, inference, evaluation, etc.), so I've never been concerned with things like function call overhead.
I work with bog-standard deep learning and this does come up, albeit not in the research stage that most people are familiar with. The closer you get to deployment, the less adequate Python becomes and the more you struggle with artificial limitations like the GIL. https://news.ycombinator.com/item?id=20301619 had a good discussion on whether we've collectively "overfit" on this slow glue + fast matrix accelerator model.
>At the end the engine that compiles the mathematical expression to hardware is what matters,
This is becoming increasingly outdated as significant parts in scientific machine learning require parts to be written that don't simply compile to GPUs. Think, for example, when you mix a physics model, say for RF or some such, and deep learning to model parts of the function. In python, you cannot write the RF model because python is vastly too slow, so you're forced to write the RF model in something fast, like C/C++, then integrate that to python, then integrate that to your favorite tensor network, needing more languages than you can do immediately with Julia.
Deep learning is moving rapidly out of simply being a tensor engine, and being a tool in much larger problems, where many high performance pieces need developed. Julia is light years ahead of Python for these domains, and I cannot see Python ever catching up because it suffers from performance and/or multiple language problems to solve these.
If you've never learned about scientific machine learning - go read some or watch some videos. It's fascinating and growing rapidly.
I’m seeing just the opposite as you: if it’s good enough to do protein folding for Google and video classification for Tesla, it’s good enough for me.
I’m sure that Julia is better for some specific tasks, for example phyisical simulations, that are important for some types of scientific tasks, but I don’t see any valid argument on why a simple tensor engine in Python is not smart enough to simulate the human brain, which is just a bunch of connected neurons.
This comment seems a bit odd given that the connections between physical neurons are qualitatively very dissimilar from the connections in artificial neural networks, so invoking the human brain seems like a red herring in discussing AI methods.
>if it’s good enough to do protein folding for Google and video classification for Tesla, it’s good enough for me.
That's funny, since protein folding uses the methods I described to achieve it's results. Multiple stages in their work [1] uses physical modeling, not NNs, to do protein folding, most likely because the problem becomes currently intractable to solve with simple Python + TF. NNs are a step to adjust the physical model, exactly like I described above.
Simply read their paper and note all the physics models incorporated at about every step of the process to enforce physical constraints - this means vastly less parameters, less training time, less training data, faster evolution of the process, etc.
Here's their repo [2]. They did that work in Python and TF, likely because it was started years ago. As Julia becomes a much faster develop tool for this type of work I expect this will change. They also used TF 1.14 - showing the age of their development. They did not release the feature generation code which is a significant component; the released code only works on the specific dataset they provide. This is likely because this component is not simply simple python they wrote, but an amalgam of things written to make the physics parts of the chain fast enough. But they don't clearly state either way.
Also, by your argument, since it's possible to solve any NN problem with a network only 3 layers deep, why not just claim that's all one needs? Because it's also not computationally feasible.
The point is that by adding outside knowledge, such as physics models, you can have vastly smaller networks, require less training data, train and infer faster, with the end result of being able to solve a much larger class of problems efficiently.
So yes, you can do it in python, or any language, if you want to waste orders of magnitude more effort and resources to do it, effectively limiting the things you can practically do.
This is why Python incurs a unnecessary cost for such development.
By willfully ignoring learning about these methods and being ignorant about even the results you cited you will miss out on extremely useful knowledge.
> I prefer having a Python monoculture for AI to be able to mix and match different algorithms
Given how divergent the current crop of ML frameworks are, is this really a realistic expectation? Having played around with Julia and Flux for ML, I find I have to do just as much rewriting when translating e.g. TF -> Flux as TF -> PyTorch. You get some limited mixing and matching with Caffe2 <-> Torch and TF <-> JAX, but that breaks down the moment you leave a company's walled garden.
> I don't think that LLVM IR that uses is the best IR for the optimizations.
I think Chris Lattner agrees, which is why he also helped start https://mlir.llvm.org/. If anything, I predict we'll see more frameworks targeting it (prototypes for Numpy, PyTorch, TF and general XLA already exist). This implies that languages that target LLVM now will actually have a leg up because their compiled semantics can be more easily lowered to something accelerator-friendly.
I agree that there is poor interoperability between the different DL / auto-diff frameworks in python. But I suspect the GP is referring also to things like scikit learn, numpy, plotting tools, PIL, opencv, pandas, etc. that make up the python ecosystem. I know that alternatives exist in other languages, but I don't know of an overall ecosystem that is as mature and interoperable as in python.
It may not be quite as mature, but it's getting there quickly.
It's also far more interoperable because of Julia's multiple dispatch and abstract types.
For example, the https://github.com/alan-turing-institute/MLJ.jl ML framework (sklearn on steroids), works with any table object that implements the Tables.jl interface out of the box, not just with dataframes.
I was using Julia before getting frustrated of how hard it is to use cutting edge AI on it, and decided to move to Python (fast.ai in specific, but moving more to PyTorch, as fast.ai 2's hook system made things much worse than if it would be something simpler).
I don't want mature, I want to use cutting edge AI algorithms and data pipelines on the newest NVIDIA GPU, so to make me move to Julia, the researchers have to move first.
At the same time for my non-AI data processing work I'm staying with Julia.
To be clear, I'm a big Julia fan and would love to be able to move to Flux.jl (I have not looked at the framework you linked but will now). But Julia is still overall more fragmented, sparsely supported, and not as easy to work with as python for ML projects. If I have a budget to work against, I cant justify trying to deal with the rough edges in Julia.
I would've assumed so too had they not mentioned both AI and algorithms. That to me strongly implies deep learning frameworks. Also, stuff like https://data-apis.github.io/array-api/latest/ exist because of how many inconsistencies are present among related libraries in the Python ecosystem.
> I prefer having a Python monoculture for AI to be able to mix and match different algorithms.
Julia is like, the poster child for being able to mix and match and compose code and algorithms: Flux/Zygote can do auto-differentiation on the entire language without modification and you can drop in quite literally arbitrary Julia code as components of your network and it works.
> don't think that LLVM IR that uses is the best IR for the optimizations.
What makes you say this? They community has been able to do some pretty amazing things performance wise-e.g. pure-Julia implementation of BLAS/LAPACK reaching and in some cases exceeding performance parity, plus there’s been plenty of work in CUDA support for arbitrary Julia code, which is impressive.
Every time I see a post, about TensorFlow for Swift.
There seems to be so many misconceptions about why this it was created. The top one being, oh it's Chris L's baby.
As a Software Engineer, who's been working on teams for the past 15 years and seeing the craft devolve, and the market saturate with people whom don't understand that fundamentals but memorize the frameworks, you need a language like Swift to bridge that gap.
It's hard enough to get people to program to an interface, let alone communicate what they are gonna return from a function or how a function behaves. 70% of people don't understand software engineering is legitimately just plumbing, they think its pretty much an artistic endeavour with no rhyme or reason. This makes it pretty difficult to stay on the same page, when building a product to scale up.
Having a strong type system solves that problem. When working with ML people my question 100% of the time is what does that function, return what type?, so I can build off what you are doing, or even debugging requires understanding of the type and the values.
It seems like team orientated programming is foreign to most people.
Swift has a low cognitive load pushing you to solve problems, with Software Engineering and reducing communication lead time between engineers.
All languages eventually converge Swift, I think Chris L has solved the language UI problem.
There is nothing special about Swift that ML languages, or .NET eco-system didn't offered already, Chris L might be a very good technical person, with several achievements, however Swift without iOS is just yet another ML/Modula-3 language with "modern" syntax.
Most people regret that Google did not invest in Julia but, to my mind, it misses the point.
Swift for TensorFlow's unique selling point was bringing a strong type system and ML+autodiferentiation together. That, for me, would be a huge boost to push ML code to production quality.
I love Swift and it's potential for use cases beyond just iOS ecosystem, but I think Apple never fostered or encouraged the open source community to involve in its development.
It's another classic example where it's simply not enough to open source something, you have to really allow and encourage the developers to contribute to increase the adoption.
I was active in the Swift language community for a couple years, and I think it was not just a lack of encouragement, but rather a sense of neglect of use-cases outside the Apple ecosystem.
For instance, maybe it's fixed now, but for years running the Swift REPL on linux would spit out a few error messages every time you ran it. It still worked, but it gave the impression of being poorly supported.
What really killed it for me was the rollout of features like FunctionBuilders (now result builders) and property wrappers. These were just basically crammed into the language in a half-finished state with no community review, to support the requirements for SwiftUI, despite many other features languishing for years and not being implemented due to concerns about how they would affect swift's "design space".
Following that, I had the impression that Swift was and would always be prioritized towards Apple's goals, and any use-case outside of this would be a distant follower.
I'm excited about a few projects bringing Swift to other platforms. SwiftWasm is surprisingly painless to get working [1]. Tokamak is an early stage SwiftUI-compatible framework for building web apps [2]. It also has the goal of supporting other native platforms and to that end there is early work on a GTK backend. I have no affiliation with these projects, though I would like to contribute to them at some point.
Theoretically if Swift was an extremely marketable language across many domains, there would be more qualified developers available to work on software for Apple products.
Apple is one of the most successful organizations on the planet, and yes they have plenty of developers in tow. I am just trying to give an example of how it could help their bottom line to push open source swift more aggressively.
If I had to guess, they don't do it because:
1. they don't think the resources it would take would represent a good ROI,
2. and/or they think it's good for them that developers who invest in their ecosystem have skills which are not transferrable to other domains
S4TF from my (external) perspective was always more a (very cool!) programming language research project than anything else, at least since Lattner left.
I would personally assume the shutdown was due to a combination of reasons:
- There simply being no good reason for Python users to ever move to Swift. There is no big painpoint being solved for the broad ML user community
- Organisational momentum lost with Lattner leaving
- General disarray of TensorFlow as a project, the parallel rise of Jax from/within prominent Google orgs, it being the cool new thing
- There simply being no good reason for Python users to ever move to Swift. There is no big painpoint being solved for the broad ML user community
As a swift and python user, I would have been really happy to be able to use swift for ML applications. Having a half way decent type system solves so many problems. But while I can see that from my vantage point, I know for a vast majority of the ML community python is "good enough" and there would be quite a lot of inertia to overcome in that regard which is probably not realistic.
I'm not sure if the Swift type system is half way decent in terms of machine learning applications. At least not without a lot of work. The data tends to be in specialized in-memory formats that tend to be opaque to the type system. The actual matrix operations involve dimension parameters that are likewise opaque to most type systems. So you still need to run code for the actual tensor graph compilation to validate if everything is correct. Python in notebooks does that instantly, Swift I suspect has a much longer delay.
I don't see any reason the swift type system could not accommodate the types required for ML. I did a lot of work with computer graphics in Swift, which is all linear algebra also, and the type system was a huge benefit.
Also I don't see why Swift would perform worse than python in any case. If python is fast, it's because it's wrapping a C++ library. There's no reason Swift could not wrap the same library, but on top Swift could communicate better type information back to the user.
In addition to what the sibling said, a big part of the S4TF project was proposing changes to Swift's type system. I'd also point out that there are not one, not two, but three type checkers looking into tensor typing for Python. What's more, Guido is on that committee!
Which is sort of my point. If you have to rewrite or expand significant parts of the the type system anyway is there any advantage over just doing the same work for Python?
There's many other advantages of Swift over Python for ML (concurrency and performance) but I just don't see the type system as one. At least not anymore.
I mean yes, because a) there's less to change, b) you still have nice features like protocols and c) Python's type system is pretty unsound. My point was that saying ML is too complex for type systems is really a cop out, and that has been borne out by how many languages have started experimenting with more complex types for this exact purpose.
> Having a half way decent type system solves so many problems
What problems does it solve for you? When building ML applications I mean. Because that's what S4TF was supposed to be.
ML researchers produce models that are used by ML applications for a particular purpose. The code to train the model _could_ be statically typed, sure, but I really don't see what the improvement would be. It would be more verbose, less readable, have significantly more noise. Just try reading some TensorFlow C++ code equivalents of their Python API.
.. static types:
* can catch bugs at compile time instead of runtime. People get very frustrated when a silly type error brings down a long training run hours into it.
* directly improve tooling experience like code completion/intellisense, jump to definition, etc.
* make it easy to know what operations are Tensor operations.
> can catch bugs at compile time instead of runtime.
OK, but what bugs? For most use cases of TensorFlow that I've seen, if there is a type error somewhere, you will catch it long before it hits production, and if you only catch it 100hr into training, your training code is either non-deterministic or there's some other issue that is not type related. Unless we're talking about dependent types a-la Idris, I don't see how Swift's type system can help catch these kinds of bugs.
> directly improve tooling experience like code completion/intellisense
Sure, but type-hints provide the same improved experience with language servers like PyLance [0]. Not a reason to jump to a new language.
> make it easy to know what operations are Tensor operations.
Easy. Anything that starts with "tf." is a tensor operation.
Yes, I'm being a bit trite, but I was never convinced by the premise of S4TF, because I've never heard an ML researcher or engineer say "static typing will fix my problems".
My guess is that's because ML researchers typically have not spent much time with statically typed languages.
I was also once convinced that static typing was not so valuable - when I was working a lot with js, python and ruby, but the more time I spend with static typing the more I like it.
There is an "activation energy" to overcome with static typing: when you first start it feels like a tedious burden to have to add type annotations, when your python code ran just fine without them. But in my experience, with a little bit of practice the type system starts to feel completely effortless, and it saves you from all kinds of strange runtime issues you run into when nobody is checking your types.
Also type systems encode a lot of intention into the code. If I look at a function signature in Swift, I know exactly what has to be passed in, and exactly what will come out. In python, I have to rely on documentation, or even maybe I have to dig into the source to see what this function expects.
In my experience, the real difference is seen when you come back to a weakly-typed language after working with types for an extended period. Returning to Python after spending time with Rust or Swift feels like walking down the street with one of my senses missing. There are whole categories of dangers I now have to watch out for which the compiler would have caught for me automatically. It is not pleasant to go back.
I've done a lot of statically typed code and have gone to conference where Haskell was on the keynote. I've done ML work in both dynamically and statically typed languages. For the ML work I do I don't honestly see the advantage. Most of the complexity is in the matrix and data operations which existing type systems don't check. And the type annotations and IDE support is good enough to catch most of the other boring errors. In my experience most deep learning ML models are also not particularly complicated code from a code perspective.
I am totally with you on statically typed vs. dynamically typed languages _in general_. I'm not denying that there are major advantages to using one for most kinds of programs. But I just don't see a benefit for ML.
You can easily find out if you like it or not, you can use the C++ API for TF/PyTorch and you'll have a statically typed program running your model for you. Sure, it's not Haskell, but C++ is statically typed, and although it's not the most popular or loved language on HN, you can use it to get stuff done.
It's awesome to see this whole comment section be about Julia! As a community, evebtually we'll realize that we dot have the bandwidth to keep inventing the same thing over and over just so a new technique can be used efficiently. Currently, Julia is the only language that makes this kind of deep reuse possible by bein efficient through just-ahead-of-time compilation that specialises well through open multiple-dispatch.
I'm curious which of the many "evolutions" that went into Swift in the last few years were motivated heavily by its dance with TensorFlow. If there were any, can we take them back out? Now that the marriage is over?
One of the things that frustrates me with a lot of the languages I'm having to grok these days, is they often lack consistency, because it's one carrot after another to get buy in from different sub communities. And in the end it tastes like vegetable soup--edible, but not tasty (if you love vegetable soup, this reach of an analogy probably won't work for you).
I think this is a great observation, aka too many chefs. My criticism of Swift, is that it has become a “kitchen-sink” language over the past seven years (I’ve been using it since the first public beta), like C++. So many features have been added that it’s gone from being a “I can keep this language in my head” language to a “I have to routinely lookup how this thing works even though I program in Swift regularly” language. It’s gotten too big IMHO and could have used a good editor instead of design by committee. The recent function builder and property wrapper additions (added it seems because the SwiftUI folks wanted them) are particularly emblematic of how it has become less readable.
The recent language additions have made me much less enthusiastic about it because like C++ it’s becoming a language where each person uses their own subset and style, instead of having a common elegance.
I remember an interview [1] with him a few months back where he seemed to have stopped pursuing it after Chris Lattner's departure. Sylvain Gugger, who was pretty involved in fastai and prototyping a Swift port, also left to join Huggingface a while ago.
He didn't just mentioned it, he thought the courses of an iteration of fast.ai in Swift as it was allegedly the way forward and more useful for students (paraphrasing).
I think people are deceived by Swift's syntax. It looks simple by syntactic sugar, but it's not actually any more approachable than other systems languages like C# or Java. Given that a lot of work in ML is exploratory, it doesn't seem like a good fit compared to a hackable scripting language like Python. I would bet against SwiftUI for similar reasons.
The syntax makes sense in an Objective-C setting because it was matching Smalltalk. It is a very nice syntax for object-oriented programming. But for functional programming it is a terrible syntax, and I think Swift kind of missed the boat on this. I think they should have pushed the replacement to Objective-C into a far more Smalltalk oriented direction as that is what the ecosystem and syntax was really tailored towards.
I do a lot of functional style programming in Julia today, but I also love how Smalltalk works. There is a place for both styles. Swift IMHO is the worst of both worlds. This very OO style syntax mixed in with functional programming makes everything kind of messy. I easily get confused when looking at Swift code for this reason.
When you use anonymous functions, you don't want named arguments. That is an idiotic idea. However if you want to pass a message to a random object, then named arguments is quite nice.
I feel Chris Lattner is not a guy who could really appreciate the Smalltalk heritage of Objective-C.
This seems like good news for Julia. Some wondered why Google would develop S4TF with an LLVM backend when Julia already existed and had better support for things like linear algebra than Swift had.
It seems this was written on the wall once Chris Lattner left the project. It's a shame since this could have been a real breakthrough in machine learning combining performance and ease. But the success of JAX means Google probably doesn't feel the investment is worth it.
I mean, I'm fairly certain that this project is the reason Chris Lattner joined Google; it was somehow in his contract somewhere. He just found a way to shoehorn himself in there and make sure that the language that he designed (Swift) is used for something more than just iOS apps. Nothing against him at all, I think most folks would want to push their brainchild as far as they can, but that's just what it looks like from the outside.
Since ML has been a large rage for the last few years, and TensorFlow is one of the most popular frameworks that ML engineers use (although I use and prefer PyTorch) it seemed like the "perfect storm" for Swift to break out of the iOS mold.
But it was clear to me (and many of my peers) that while this was an interesting project, the need for it is really low, and that nobody is going to throw away years of experience with libraries like TensorFlow and PyTorch (via Python) in the trash just so we can use Swift, which for me isn't really that crazy of a language as all the hype makes it out to be, and I'm sure the same is true for many ML devs and researchers alike.
I'm looking through the meeting notes[1] linked from the archive PR[2] and it looks like there hasn't been much activity on the project recently. Can anyone with more insight provide an explanation on what happened?
My reading of the discussion at https://forums.swift.org/t/swift-concurrency-roadmap/41611: the Google Swift team members expressed hope for a bit more community involvement in the Swift language planning, and Apple folks very politely told them to shove it where the sun don't shine.
My guess: Google concluded that investing in a language developed behind closed doors of another company is a bad deal.
I don't blame Apple, it may be a good business decision for them, and of course it is completely within their rights.
So that's it then - another decade of unchallenged dominance of AI by a Betamax scripting language designed by accretion for early 90s CPU architecture. I just don't get the whole Python for ML craze.
Swift was a weird choice for statically typed TensorFlow, being only
popular on the platform, that does not have GPU/TPU support in TensorFlow,
which is, basically, a requirement for any serious work. The fact, that
they had to fork the compiler did not help either.
TensorFlow for C# is much like TensorFlow for Swift. There's a statically
typed binding to the core types with the rest of the TensorFlow API available
through an open source Python.NET project [1]. Unlike Swift version though,
that second part is also mostly statically typed, so you can get IDE
autocompletion hints. Also, .NET runtime has native support for dynamic
languages.
Like with Swift for TensorFlow, we ported all recent interesting
neural network architectures (and maybe even more): Transformers (GPT-2) [2],
CNNs (YOLOv4) [3], Q-learning (e.g. RL, actor-critic) [4], and even some
cool ones, that have lots of unexplored potential like Siren [5] (this one
uses sin/cos as activation functions).
Although we have not worked on automatic differentiation yet, unlike
Swift, it will not need compiler support. .NET (and Java) can inspect and
generate code at runtime, so autodiff can be implemented in a library.
We also have integration with Unity ML Agents for training robotic agents [4].
> Swift was a weird choice for statically typed TensorFlow, being only popular on the platform, that does not have GPU/TPU support in TensorFlow, which is, basically, a requirement for any serious work. The fact, that they had to fork the compiler did not help either.
This is available now (not sure if it's in mainline).
Do you know if tensorflow for Java is as mature as tensorflow for C#?
Btw of you didn't saw it, Facebook is implementation support for auto differentiation to the Kotlin compiler
This is sad for the project itself, but I predict that the coming programming competition will be Rust vs. Go, not Swift -- as Swift was always a distant third among the "new" languages. And of course the more mature languages will continue to be popular and widely used for decades.
In my opinion, Rust will end up winning this contest, but others have perfectly good arguments against that view. Unfortunately for server-side Swift and the many good people who've worked on it, there seems little chance for it, in TensorFlow or other usecases.
The idea there's one language for everything was false even during C's heyday. Your list contains languages for very different uses.
Rust will eventually win low-level programming from C++, but massively networked applications will always be slightly easier in Go. Swift is a bytecode language, it competes against .NET (mainly C#) and JVM (Java and variants), not Rust or Go.
IMHO Swift seems to losing when not tied to Apple's ecosystem, its future will be as some Apple-niche language.
I guess it's all in how you classify it... I'm not sure why "bytecode" vs. "non-bytecode" are an important distinction in the language's usecases, as opposed to implementation. For example, .Net supports both static compilation and bytecode interpretation, as do most LLVM-backed languages.
Maybe the better definition would be 'GC languages with a runtime'? There's a distinction between languages that require a runtime and impose a GC (with the associated penalties) and 'low-level' languages like Rust* .
I'd say 'GC runtime languages' are somewhat dubious for low latency etc. (but see Unity and ZGC!), while 'low-level languages' are overkill for typical UI/CRUD apps, I don't see any useful advantage for Rust there.
* Go would be in the middle I guess - technically requires a runtime, but the runtime is relatively light and all binaries are static - unlike .Net whose static compilation support is frankly rather theoretical right now due to common use of its powerful reflection capabilities.
that's obvious to me, what sounds more like a prediction is kotlin to be a dominant application development language (fullstack/scripting) natively compiled or on the JVM
It was Chris Lattner's baby and when he left Google no one cared to push it forward. The Google vs. Apple fights over ads and privacy probably didn't help the case.
In this case it's not just a library like the wrappers for tensorflow already present in many languages (including python), it's something like a Swift language fork in which the language constructs are analyzed by the forked compiler during compilation in order to directly generate an equivalent tensorflow execution graph (without manually creating a tape through library calls). Thus it's a Swift (compiler plugin) for tensorflow.
The intention was to merge the fork into the main language making Swift differentiation tools first class within the language.
I had the same question for the naming of Windows Subsystem for Linux (WSL), which IMO should be named "Linux Subsystem for Windows"
The best explanation I can think of is that the people naming it wanted the first word to emphasize the thing that's most important to them. In Microsoft's case, that's of course Windows. Perhaps the same is true for the S4TF folks.
I am disappointed, but so it goes. I started writing a Swift AI book a while back and I had planned on abut 20% of the book would be on S4TF. I still have lots of other material, but obviously I am not going to include S4TF material.
BTW, the Haskell bindings for TensorFlow have been actively maintained for years. The latest update was 3 days ago.
It is much easier to maintain a language binding than a full toolchain. There are a lot of activities in Swift mainline, and I can only imagine how much work it must be to maintain the S4TF fork.
> Added language-integrated differentiable programming into the Swift language. This work continues in the official Swift compiler.
While the readme tells us that the project is in archive mode, it also tells us that the differentiable programming part is still worked on by the compiler team.
Is there further information on that?
It's kind of odd. Chris had a huge impact at Apple with LLVM and Swift still going strong and forming the backbone of the developer tools. But I'm not sure he has anything of lasting value to show for his jobs since then. Maybe evidence that for great engineers, just like other great people, fortuitous timing and circumstances are a big part of the story.
I was using this project for my stock trading bot.
Way nicer to use Swift than python for this, and I have rewritten this bot several times in different languages + frameworks (objc php java python swift tf1.0 theano keras s4tf) since 2012.
Doing authentication for brokerage apis + correctness from pulling from the db is easier to get right in Swift than python bc of the static typing + grand central dispatch for parallel queries.
It is also way more productive for me to have the compiler catch issues at the beginning instead of 6 hours into a data generation job where the python interpreter sees a None for something I didn't think of.
Not surprised that this project had trouble gaining traction given Swift's obsession with shoe-horning functional concepts into every crevice of the language, e.g., I can't get through a single Swift tutorial without having to get into closures.
Closures aren't really a functional concept, they can just be used functionally. You'll see a big reduction in closure usage when the new concurrency features ship.
I am not involved in data science at all, I have extensive Python experience, and did some work in Swift for a couple of years.
Putting aside the Python 2/3 transition, I think that Python does a good job of introducing new features carefully. For example, the "X if P else Y" syntax. While I prefer the C approach, the Python approach works fine and fits in the language well.
I had a mild dislike of Swift from the beginning, but it was a lot more palatable than working in Objective C. What bothered me was all the gimmicky syntax. Sometimes "let" unwraps an optional, sometimes it doesn't. ! and ? seem badly overused. ARC doesn't compose well with other language features. There is a ton of syntax whose only purpose appears to be to stuff logic into and around assignment syntax. Then it got worse, with major language changes across versions 3-5, each one introducing lots of new concepts and syntax.
And of course, Swift is a proprietary Apple language, (in the same way that C# is tied to Microsoft). When I stopped doing iOS programming, I very happily but Swift behind me.
I saw the TensorFlow additions to the language and thought the whole thing bizarre. I put my reaction down to my lack of involvement with data science. Maybe I'd appreciate those changes more if I knew more about data science. But I also remember thinking that it was quite odd to make such extensive additions to the language to support this one application area, which was very, very far from the language's sweet spot.
This is hilariously wrong. Actual fans of functional programming would disagree with you, as there are several things Swift doesn't do to enable pure functional programming.
Swift's a delightful language to use. It has a lot of the nice things about Rust's type system, but is a heck of a lot easier to use at the expense of a bit of performance. For a lot of use cases, I think this is a great value proposition. ML/data science is a great example where practitioners could benefit from a more grown-up type system than what Python has on offer, but would still be able to keep low level details at arms length.
I think Serverless would be another ideal use-case for Swift, where the productivity, clarity and correctness tools it offers would be a huge benefit.
Some very interesting things came out of the S4TF project - like the work on autodiff, and python interop. It's a shame the project never really seemed to get legs, it seems like it just languished for years with no clear direction.
These days I do most of my work in Rust, and I'm happy to do so because the tooling, community, and ecosystem is really amazing. But there are a lot of language concepts and features I miss from swift. I guess it goes to show that the governance and availability of a language have a lot more to do with adoption than the merits of the language itself.