I have a pretty similar background. I have an undergrad in ChemE who fell in love with machine learning research. As I didn’t had the appropriate background so I taught myself Computer science using mostly resources such as OCW and teachyourselfcs, videolectures etc.
However, what stood out to me was how difficult it is to self study? Universities provide a setting which helps you learn difficult subjects over a longer period of time. Outside of that, no such avenues exist. It’s not just reading up a book or making Anki flash cards(which is quite tedious to be honest) but the process of selecting, vetting what to read next and actually completing it. Or the question are we there yet, with no idea what “there” actually means.
I decided to work on creating a platform where people can actually learn difficult subjects on their own. It’s a bit different than normal video based platform, it actually uses text based conversation to facilitate learning. But I truly believe this is the way forward.
I have written comic based guide about this problem:
I would love to self study with more vigor. I studied Mechanical engineering, but I find Math to be so beautiful. I think I chose the right education for the work I find meaningful in the world, but I wish I had more time to learn math that isn't directly relevant to what I am doing.
I'll certainly check out your site and save it for when the world is more peaceful.
Wow that's what I was gonna write. I did engineering too, and all of the topics listed in the course. But something about doing it to apply to other things is wrong, hard to say what. I remember in high school being quite fascinated by certain beautiful things in maths, and the applied stuff just isn't the same. I guess it just seems like tools. All the theorems are just kinda unsurprising, as far as I recall. Things like Stokes Theorem are kinda foreshadowed as you're doing your reading. Or various statements of continuity (DivGradCurl).
Everything that was discovered in math and physics was foreshadowed at the time. Your description means you were being taught correctly by excellent knowledgeable people. If when you moved onto deeper subjects this was no longer the case, it means the learning regime you were undergoing wasn't optimal. This is especially true for pure math. Pure math is an experimental science like any other. It may feel more mystical and interesting when you are learning stuff that seems to exist for no reason, but really you are just missing out.
> However, what stood out to me was how difficult it is to self study? Universities provide a setting which helps you learn difficult subjects over a longer period of time. Outside of that, no such avenues exist. It’s not just reading up a book or making Anki flash cards(which is quite tedious to be honest) but the process of selecting, vetting what to read next and actually completing it.
One major thing I found when learning at a university that I have trouble with when self studying is pacing.
Let's say the textbook covers some topics A, B, C, D, and E in that order. When self studying I start with A, and keep working on A until I'm confident and comfortable with it, then move on to B, and so on.
In school we'd spend a while on A, but then usually move on to B before I was fully comfortable and confident with A. Same for B to C, and so on.
This was fine, because while I didn't think I was ready to move on the instructors had a lot of experience with teaching this material and knew that we students knew enough A to understand B, and in fact that exercising our A to help us learn B would also strengthen our A.
This is definitely one of the hardest parts of self study and what gives me enormous appreciation for good teachers. Progress can slow or even halt because you are forced to be so rigid about understanding everything before moving on- otherwise you risk serious gaps in knowledge. I have found CS classes with autograded programming assignments far better than other types of online courses because I known if I at least learned one thing.
Excellent intro comic, it got me really excited about your product! I would love to try it out once you release a more advanced course. I signed up, hopefully you will send e-mail updates as you release new courses.
This is very well done. I've been hobby researching learning tools for the past couple years and this looks like one of the best. @ai_ia have you seen Andy Matuschak's Orbit project?
I self study machine learning here https://learnaifromscratch.github.io/ai.html it's an early and shitty draft and proof of concept that you can do self-directed learning for these topics while looking up the background you need to know, which for me is much more interesting than taking a generalized math curriculum of absolutely everything. The courses so far we haven't escaped the content of Wasserman's 'All of Statistics' book yet on classification or probabilistic graphs, so you could if you wanted watch the lectures and only do Wasserman's book.
If you want to try the OCW linear route of taking everything for whatever reasons, you will have to get up to MIT student levels trying to unravel the algebra done in the early calculus courses and later where they just assume you possess this background. One way to do that is those problem solving books like this one which 'bridges the gap between highschool math and university' https://bookstore.ams.org/mcl-25 at least you then get worked out solutions. Another way is Poh-Shen Loh's Discrete Math course he opened up on YouTube which is done the same way he holds the CMU Putnam seminar, working through a bunch of combinatorics and algebra will more than prepare you to understand those continuous math OCW courses https://youtu.be/0K540qqyJJU
Like everybody else there is of course the issue of: who is going to check my work. For me I went with the time tested tradition of hiring a tutor, a local grad student and paid them once a week to go on chat/zoom or meet at a coffee shop before the pandemic and spend a few minutes going over everything I'm doing wrong. In the early days however I used constructive logic ie: 'proof theory', to audit my own work: https://symbolaris.com/course/constlog-schedule.html and read a huge amount of Per-Martin Lof papers on the justifications of logical operators like implies, disjunction, conjunction, etc. Of all the math I've ever taken I would say that proof theory was the most useful for somebody by themselves who isn't sure of what they are doing (I'm still not 100% sure.. hence why I hire people now).
If you want a great Calculus text that explains those nasty looking Euler's e nested statistics distributions try Mathematical Modeling and Applied Calculus by Joel Kilty everything from partial derivatives, gradients, x^n, e^x, trig, integrals, limits is explained in terms of parameters to modeling functions, if you write software it will be easy to understand. I haven't posted it yet but I tried going through Allan Gut's probability book using only that math modeling calc text and have not run into anything applied, as in concepts about limits or integrals, that wasn't already covered. Of course the concepts are much more abstract measuring a bunch of intervals and a different method of integration and I don't pretend I'll be making any advances in this area beyond applied usage but it can be done, jump in and pick up the background as you go as opposed to doing all the background at once, losing interest and giving up.
If anyone is interested in following a self-study curriculum but with more of an emphasis on pure vs applied math, you may find my experience and recommendations useful:
I’m doing something similar, except for Stats. I’ve cobbled together a plan based on degree programs from Stanford, CMU, and Berkeley. It would seem easier to stay on track with directed course learning, but how do you stay on track with the self-directed learning?
Is this doable for someone with less basis in math? I almost stopped studying math after two years of undergrad, when I went to a more practically-focused school. It'd be nice to get back into shape, because I've started reading research articles and always feel like I'm behind on the theory side of ML.
It depends very much on your own skills and drive.
At some point, you need people to check your work, though, but you can search for online communities to do that. E.g. a post to mathoverflow asking about whether the following proof is correct, for example. Try to find a community of others online and work with them.
Also remember that math is about ideas. It is not just an exercise in formal deduction. The field is dense with non-trivial, non-obvious, interesting insights, and understanding these insights so well that overtime they become obvious to you is what it means to learn math. Sites like 3Blue1Brown to a good job trying to explain these ideas, but they really put a lot of work into it. Most math texts do not, but you have a professor or classmate you can talk to -- so there is a gap between the text and what you need, but perhaps you can close the gap with online resources.
> At some point, you need people to check your work, though,
Agree with that, 100%. Most of my students will, at some point in a course, respond to questions with things that are completely wrong. It's just how people learn. If there was not someone to correct their mistake then they would struggle a lot to find the error and fix it.
One of the most rewarding things about working with students is helping them through tough spots. But it is real work, and takes the time and attention of someone who has training.
> but you can search for online communities to do that. E.g. a post to mathoverflow asking about whether the following proof is correct, for example.
I'll just observe that MO, while a great community, can not scale to helping lots of people check their homework. For instance, in a Linear class session I might have 20 people, each with 12 homework questions. MO is just not set up for that.
Now, sometimes students are sure they got questions right, and usually their certainty is right, so there is no need to ask that one online. And sometimes people can find a relevant previous answer (although learners often struggle to find those, in my experience). Nonetheless, even after taking those out, there are still a lot of people with a lot of questions.
> Try to find a community of others online and work with them.
Yes, very good advice. I just wanted to observe that while it might work for OP (and I hope it does), it cannot work for lots of people. I don't know what the answer is, and maybe MO could help a lot, but it can't be the answer alone.
Yes, I agree with all these points. I wasn't trying to suggest that access to knowledgeable teachers isn't necessarily, but giving second-best options to those for whom that's not an option.
The issues will be the same with self studying anything. Without someone to critique your work, and point out deficiencies you don't even know are a thing, continuous improvement quickly becomes exponentially harder from the bad habits that are holding you back.
If anyone wants to attempt this, that's awesome. It's a lot of hard work, but I think it opens a lot of opportunities for personally and professionally. I've a Ph.D in Applied Mathematics from a traditional program, so I wanted to chime in based on some of the comments I'm seeing. As a note, this is an opinion and others may feel differently and strongly at that.
To me, applied mathematics is the art of transforming something into a linear system, which is something that we can tangibly solve on a computer. There are lot's of ways to do this such as Taylor series and Galerkin methods, so a lot of the field is understanding how, when, and why each method can be used. This is coupled with mastery over linear solvers, which includes direct methods, iterative methods, preconditioners, etc.
I wanted to write this comment, though, to focus on certain areas that may end up blocking what I view as appropriate progression in the field. These are things that I believe are necessary to understand advanced topics, but don't necessarily fall under applied mathematics. First, you really do need mastery of real analysis. It's necessary because it covers formally topics such as differentiation, integration, and series, which are required to understand theorems and algorithms. In my opinion, calculus books are not sufficient. Rudin's Principle's of Mathematical Analysis is the most concise, well written book that contains enough. Second, enough functional analysis to understand Hilbert Spaces is required. This prerequisite to this is the real analysis above. The issue here is that algorithms for things like differential equations require function spaces to do properly. Certainly, you can go really deep in this regard, but Hilbert Spaces are generally enough for practical algorithms. This also affects optimization theory, which impacts machine learning. Technically, you can do optimization theory with only real analysis, but the theory is cleaner in Hilbert Space. Questions that need to be answered are things like does the infimum exist and can it be obtained? Working with a general inner product is also a valuable tool for parallelization as well as a modeling tool. Third, some integration or measure theory is required. It depends on what you're doing, so I don't think mastery is strictly necessary, but spaces like L2 don't make a lot of sense unless you know what a Lebesgue integral is. Even if you want to just work with spaces that are Riemann integrable, measure theory helps understand when this is possible and the ramifications of it. And, to be clear, this is important outside of differential equations. If you want to understand optimization theory in a Hilbert Space, the inner products used will require some understand of measure theory.
Anyway, these are some random thoughts and ideas about the field. I do believe strongly that any amount of study is beneficial as most engineering fields benefit from applied mathematics.
As far as books go, we used "Real Analysis" by Carothers for analysis in undergrad at Caltech and it's one of only two math textbooks (the other being Dummit and Foote for algebra) that I go out of my way to recommend, particularly for self study. It's probably not a sufficient book if you're pursuing a Ph.D. or anything, but I would definitely not start out with Rudin (too little hand-holding, you'll die), and Carothers was amazing.
I recommend anything by George F. Simmons, as well as anything by Russians. Seriously, if it's a Russian sounding author's name, then odds are good that it's well written.
They had amazing pedagogy and the EMS series (a joint publication by Springer and the old Soviet publisher VINITI) is first rate.
Thanks for providing another suggestion on the real analysis front. Really, I agree with you on the Rudin front.
Honestly, I think that real analysis is the largest barrier for people trying to self study applied math. I'll mention here that I actually had a total of five semesters in real analysis all said in done. The first was an intro to real analysis course that used "Introduction to Real Analysis" by Bartle and Sherbert. The second was a more advanced real analysis course that was two semesters long using "Real Analysis and Foundations" by Krantz. When I entered my Ph.D, I was given an initial evaluation and the committee found my real analysis still too weak, which necessitated another two semesters using baby Rudin. Only at that point, did it really take and it's not been an issue since.
Anyway, mostly I wanted to echo that I agree with you. I think Rudin is the metric to say good enough, but I was not able to get through it on first go.
Laugh. I was hoping no one would ask since I don't have a great answer! We used "An introduction to Hilbert space" by Young. It's fine? There's stuff in there like Sturm-Liouville systems that I don't particularly use or care for. I also never use it as a reference.
Here's a hodgepodge of other books related to functional analysis that I like more, but don't directly answer your question. I got a lot of benefit out of "Convex Functional Analysis" by Kurdila and Zabarankin. They sort of have a high level overview of different functional analysis topics, including Hilbert Spaces, but with the ultimate goal of proving what they call the Generalized Weierstrass Theorem. Essentially, when does a function has an inf and when is it attained. Even if you don't care about optimization theory, I very much appreciated their survey of topics to get there.
I occasionally also use "Introductory Functional Analysis with Applications" by Kreyszig as a reference. I think this was the first time I saw cleanly the difference between an adjoint and the Hilbert-adjoint of an operator, which was constantly confusing to me prior to that point.
The last one I like is "Nonlinear Funtional Analysis and its Applications I: Fixed-Point Theorems" by Zeidler. He wrote, I think, five volumes, but this is the only one that I use. Anyway, he presents differentiation, Taylor theorem, and the implicit function theorem very well in function spaces. The first four chapters are great as a reference.
Since I'm listing off obscure books, for integration, I like "A Concise Introduction to the Theory of Integration" by Stroock. I actually don't like his newer book, "Essentials of Integration Theory for Analysis" as much as the older book. Anyway, I find it very dense, but well written. Essentially, I like the first five chapters, which culminates with the divergence theorem, which ultimately gives a precise description of integration by parts in more than one dimension. He also answers precisely the question about the difference between Reimann and Lebesgue integrals.
Primarily, it gives us the power to work with functions that return other functions. In many ways, it's kind of like going from array processing in Fortran to functional programming. Outside of the esoteric, there's a choice that has to be made while modeling as to when to discretize the system into a linear system. The ability to use a function space means that we can manipulate the formulation into another form before discretizing, which impacts the convergence theory as well as the practical performance.
The reason that I mention Hilbert spaces is that they have more structure than a general function space, which makes working with them easier, but still general enough to be useful. Essentially, we get an inner product as well as the ability to enumerate an orthonormal basis, which makes it feel more like working with linear algebra.
Even in R^n, I believe strongly that it's important to code with these abstractions. First, it makes parallelizing the algorithms easier. If we treat a vector as simply an array of numbers and use the dot product, then the code requires a more significant rewrite when moving to multi-computer parallelism. If we treat them as a generic vector object and have an interface that works with inner products, addition, scaling, etc, then the same code can work either in serial or in parallel given two different implementations of the vectors and their operations.
In addition to the abstractions, the choice of inner product is important. If we have a linear operator in a Hilbert space and discretize it, we generally have to discretize three things: the operator, it's Hilbert adjoint, and the inner product. If we implement this blindly, they're not consistent after discretization. Meaning, the property that we should get is that <Ax,y> = <x,adj(A)y>, but this probably isn't true if not done carefully. Generally, we can freely discretize two out of three of those operations and then the third one needs to be adapted for consistency. Maybe you want to choose the discretization for the operator and its adjoint, but this probably requires a non dot product for an inner product. Alternatively, maybe you're optimizing some problem and realize that some of your variables are out of scale, so it's converging slowly. You can certainly just rescale your variables with a diagonal scaling. However, you can also change your inner product, which changes the gradient, which also rescales the problem in a different way.
Mostly, that's to say that an inner product should be a choice that one freely and intentionally makes. Learning to work in Hilbert spaces forces us to become comfortable with this approach.
I tried to do the honors calc sequence through OCW and I found that I really couldn't hack it on the textbook alone, I'd get ~60% on my self-graded homeworks and I'd spend weeks on individual problems. Often I'd get stuck just trying to figure out the notation, which differed from the book. And I found that the recorded lecture material runs out pretty quick beyond intro level classes.
I ended up enrolling in the math 23 sequence at the Harvard Extension School and found it to be a lot more valuable.
Just don't spread yourself out too thin, and know when to stop - it's super easy to burn out when taking on too much coursework, even if this extra part is self study.
The author writes that he wants to pursue a Ph.D - in that case, I'd put all my effort into pursuing what would maximize my chances at getting into such program. Unfortunately, self study is quite difficult to prove, and does not hold much weight when applying for such positions.
> I think that would be more likely to be covered in a pure math program than applied math.
Integration is pretty basic and used extensively, and I'd say it makes no sense to cover contour integrals within the scope of complex numbers and differential equations but leave out integrals.
Integration is covered in any calculus course, the comment I was replying to do was about Lebesgue integration which is a much more advanced topic that as far as I know is only needed for integrating functions which are so pathological that they probably don't occur in the physical world.
I am no expert here, but as I understand it, measure theory is used as the basis for all modern probability theory for the reason that it simplifies a lot of things as you get to more advanced topics, like Markov chains on general state spaces [1]. So if you want to study Stochastic Processes you probably want to use measure theory. So thus the Lebesgue integral is not just for "pathological" sets (I remember reading somewhere that even the Lebesgue integral still breaks down on some really pathological examples). If you've studied mathematical statistics, you'll typically see proofs for expectations of discrete distributions and continuous distributions. They're typically similar, but different since you're using summation for the one, and integration for the other. When you get to random vectors, you can have weird distributions where some components are discrete and others continuous. Apparently using a measure-theoretic approach to probability, it unifies these into one general theory (again, I'm no expert, this is how I understand it).
The applied math does teach integration: the Riemann integral. Knowing about Lebesgue integration is not really necessary for most of this kind of work.
This guy was my TA in a GT for a course on Educational Technology. He totally blew off giving me any feedback until the last possible moment, super frustrating experience.
There are always three sides to every story (his/hers/truth).
In the case of undergrads it can often be a case of everyone procrastinating equally and then initiating a time crunch crisis as they all ask the TA for emergency help simultaneously.
However, what stood out to me was how difficult it is to self study? Universities provide a setting which helps you learn difficult subjects over a longer period of time. Outside of that, no such avenues exist. It’s not just reading up a book or making Anki flash cards(which is quite tedious to be honest) but the process of selecting, vetting what to read next and actually completing it. Or the question are we there yet, with no idea what “there” actually means.
I decided to work on creating a platform where people can actually learn difficult subjects on their own. It’s a bit different than normal video based platform, it actually uses text based conversation to facilitate learning. But I truly believe this is the way forward.
I have written comic based guide about this problem:
https://primerlabs.io/comics/introducing-primer-comics/
We are creating self paced courses on Computer Science and also looking into creating self paced Mathematics/Physics courses as well.
We have released two free courses for everyone to try out at https://primerlabs.io