Your attitude is common, but mistaken. It's another version of "the grass is always greener". Whether you're working on a self-driving car or implementing Microsoft Word 2014, the fundamentally hard parts are actually the same.

The self-driving car might sound sexier if you've never done something like that, but once you've learned the basics of how control loops work and the quirks of realtime operating systems, that's about it for novelty. What you'll actually spend your time on is the same old stuff: managing complexity. Debugging, testing, designing APIs, designing user interfaces, integrating disparate module, etc. You may actually find that it's more soul-crushing than a basic CRUD app, because in a safety critical project you'll probably have less autonomy, more oversight/bureaucracy, and a much slower pace.

The fundamentally hard part of any software project is complexity. It's easy to let even the simplest project spiral into unmanageable complexity, and it's deeply challenging work to prevent that. That's the meat of what any good software engineer does.

There's a better way to word your original question: in what kind of organization should I be writing software? That has a far bigger impact than the problem domain on your happiness, level of challenge, and autonomy.

I think there is an exception to this for people that work on programming languages -- there the best programs written.

In my spare time I have never had more fun programming then writing interpreters for scheme, prolog, and smalltalk. It would be amazing to get paid to work on tomorrows languages!

It's not the only way but a PhD will certainly open a lot more doors at the "high end" research end of things. That said, even Microsoft Research will hire "Research Software Development Engineers" who work with researchers to implement cutting edge stuff as well as migrate it into regular Microsoft products, and only a bachelor's is required: http://research.microsoft.com/en-us/jobs/fulltime/technical.... .. these sort of roles can blossom into more challenging and more research oriented work over long periods of time.

This is a rather crude generalization, but PhDs are often demanded at high end research positions not to prove you know your stuff, but to guarantee you have the mettle to do research and bounce back from failure after failure.. a must if you're working on someone else's dime. Of course, if you can find another way to support yourself, you can do (almost) any research you like in your own time without strings attached ;-)

If I had money and free time, then I'd love to do research on my own. Thing is, I need a job to pay the bills :( I'm thinking of getting into freelancing and using that to float me as I get to do more and more research.

This is confusing to me, what is taking up your time?

The way I see it people choose what they want to spend their time doing. Sometimes they choose research, sometimes they choose game playing, sometimes they choose reading Hacker News :-) but there is generally enough time for folks to do what they want to do.

Research is especially easy if you know what you want to research. But it's not a simple matter of "Hey I'm in research, lets blue sky something!" People in research have some passion that they are following, like parsers, or graph theory, or AI, or network coherence, or something. Some area where they have lots of questions and no answers.

Keep a notebook. One for each topic if you want, they can be the cheap composition ones that go on sale during 'back to school week' or really nice lab notebooks. But they are like pitons and rope to a climber, they anchor where you have been and where you are going. Write the questions you investigating into the first few pages after the index. Start writing down what you know, what you can prove, and what you don't know. Invent experiments or ways to discover the answers to the questions you don't know.

I've got a notebook with a couple hundred pages devoted to making carbon nanotubes. I know lots of ways you can't make them and several ways you can. Sometimes you can scrounge equipment for stuff, sometimes you have to invent (did you know that you could use a bottle of propane and a chunk of copper pipe as a way to do vapor deposition of carbon? You can! And if that copper tube rolls off the stand onto your wooden deck when its hot it leaves a big black mark? :-)

"Research" doesn't need a government funded lab, it needs good laboratory technique. Careful note taking, documentation, exposition and analysis. Software research is even easier these days because of how cheap computers have become.

In general, having a job doesn't prevent you from asking questions or doing experiments (aka "research") but if you're tired after working all day and just don't want to think any more, that is a different problem. Some folks I know get under demanding jobs just so they can do it easily without straining too hard during the 'day.'

One of the things I did as a kid that drove my Mom nuts was read every scientist biography I could, I tried to re-create their experiments because anything an adult could build in the 1800's I figured I could build as a kid in the 1900's. But what it really taught me was how these people pursued the questions they had.

Bottom line, "money and free time" isn't an excuse, its a rationalization. Why are you denying yourself your own research?

This is the strategy I'm currently implementing. I've been doing freelance work while working on research with physiological sensors as computer input.

You'll need two things to get started doing this sort of thing: standing out as a member of your community so people come to you, and an understanding of business needs.

If you really want to make the switch, shoot me an email. I'll give you some tips on making the transition.

So, I can't find your email. I checked your profile and your blog. I can follow you on twitter or whatever you like, but your advice would be extremely valuable.

Sounds like you could write publicly about this, it sounds fascinating :-)

Tweeted at you. Please respond when you get the time. Thanks.

can i? i'm on the same lines as the OP.

Im channeling Peter Thiel here: there is little connection between hard and valuable. The goal of most companies is to create value, and do it without expanding too much effort.

If you really want to challenge yourself give yourself a goal like:

1. Decode the genome 2. Find the correlation between all aspects of someones lifestyle, diet, and their health. 3. Solve poverty - maybe there is an economic solution.

The world is full of hard problems, but you have to take the lead if you want to work on them.

I think employers will be interested in hiring you for such projects only if you have demonstrated knowledge and experience in the field. If your work experience primarily consists of RoR web development, most employers will pass. It's not that it's implied that if you do RoR, you won't know about algorithms and AI, but there is a high correlation, and sorting through a ton of RoR resumes to find a worthwhile candidate is too much work.

You can work on hard projects on your own. Once you have made substantial contributions, you will have credentials to interest the employers. Most of the open, hard projects have very few full time developers. PyPy has only 3 I believe. If you have something to contribute, people will be more than happy to have you on-board.

Follow the advice in http://steve-yegge.blogspot.com/2006/03/math-for-programmers... and learn math. Once you have the skills to do the interesting stuff, you start to see opportunities to do it in the oddest of places. For instance I've been paid to do things like statistics and machine learning on "simple CRUD apps".

Maybe you won't be that lucky. But if you don't have the required skills, then you definitely won't be that lucky.

Incidentally a fun place to test the intersection between math and programming is http://projecteuler.net/. Besides, it is fun.

Most machine learning work is the CRUD of math. It is plug-and-chug in standard toolkits.

Hard problems are everywhere. Every company, every app, every product has unique and difficult problems to solve.

Sometimes those problems are with product design and UX. Sometimes those problems are mathematical. If you are successful, often they have to do with scalability and reliability. If you work with great people there are always hard problems in enabling productivity and building great tools. Many companies have serious and significant challenges around security and may not even know it. If most companies are solving simple problems, then why aren't most companies automatically successful?

If you broaden your definitions of "hard" you might find that there are many things that qualify. If what you mean is "hard mathematical problems" then yes, you will have to learn some math.

Try looking for bioinformatics jobs. It involves big data where the amount of data to be analyzed is growing exponentially every year, a lack of knowledge on what to research to provide meaningful results, and huge benefit when people come up with novel ways to interpret data successfully.

It will involve acquiring some domain specific knowledge to be truly successful (what hard problem doesn't?), but you will be able to dive in straight away and make huge contributions just by making current analysis more efficient!

+1 for bioinformatics - adding to that are the issues surrounding the complexity of the data (look up HL7 or CDISC to see how complex) and also the legal standards and ramifications (HIPPA, 21 CFR Part 11) of working with clinical research.

In short, getting domain knowledge in the above are essential to solve the tough problems at the big-pharma or national-bio-labs level.

(PS to the other poster looking for bioinformatics programs: I suggest you look in Boston - Tufts, MIT, Harvard etc)

I'm slightly hijacking the thread here, but does HN have any recommendations on CS-heavy bioinformatics grad programs (not just US based)?

Threadjacking? This question should be a front page discussion, instead of gossipy crap.

I would love to know as well.

I second this. In computational biology we work on hard problems with real outcomes which potentially change or save lives. Some of the problems are probing at the fundamental mechanisms of life. We need more programmers who aren't afraid of hard problems.

I don't know, I still remember this from HN: http://madhadron.com/a-farewell-to-bioinformatics

Try the games industry!

The pay sucks compared to other development, and work hours are crap, but you will be able to work on AI, gameplay, Graphics, physics (whatever your experience is).

But in all honesty, if you started programming a couple years ago, and learned a bit of rails/ruby, you have a lot to learn in various languages and systems (try C, embedded, iOS/Android, learn Haskell, big data, etc)

It sounds like going to a Disney Land and complaining that all rides are easy. If you want tough and challenging rides go to SixFlags.

If you want to tackle tough challenges may be ruby/rails is not the ideal place. Perhaps learn C,C++ and implement faster ruby or javascript interpreter or implement a distributed filesystem in ruby or java. You can also contribute to the rails framework.

I've learned that ruby/rails is not the place to go for tough challenges. Agreed. My question now is where do I go from here?

If you like computer graphics, there are also tons of interesting problems in that field.

Lets consider Maths, as an example to answer that. Well, up till college, we are taught to solve problems that are already solved, towards our PhD, we come across problems that haven't been solved, they are hard, but they are not the end of it, beyond that lies a sanctum, where you must find a problem and fix it. Thats what entrepreneurs do, it may sound easy, but do not be fooled by it, because being an entrepreneur it seems easy when you have found a problem and its solution, or as Steve Jobs said " looking back, the dots connect but 20 years ago, looking forward, they did not"

Billions of people around the globe haven't even noticed many a problems, yet a handful of entrepreneurs did, wasn't that the first part of problem solving, and if you haven't probably noticed, you are here after all, so you might unconsciously know the answer lays somewhere around here.

So from my eyes, whether its programming or maths, the hardest problems that exist are the problems no one has noticed yet.

For your case, you might want to find a problem of programming, if you have programmed for that many years, you will surely run into one, that no one has noticed. If you haven't, as Steve Jobs also said "Keep Looking"

I'm with you. I really like solving problems or finding a good algorithm to solve the problem. What I can say for you is try to find a company with lots of R&D going. I think there are many companies out there, especially startups that are having some R&D to improve their product and beat their competition. I think startups like Prismatic (http://getprismatic.com) and Diffbot (http://www.diffbot.com) have a heavy NLP and machine learning research going in the background to improve their product. I'm interested in joining those companies I mentioned and contribute to their reaserch and development.

I think game companies also have an R&D to improve their AI, path finding, rendering algorithms, etc.

That's it. Just find an R&D job.

Game engines are hard. Working on CryEngine, UnrealEngine, idTech7, or something equivalent sounds very challenging. Games involve AI, physics simulation, concurrency, high performance, and more. The salaries is not that great in the game industry though.

"CRUD" is hard. That's why there are so many bad CRUD applications out there. Write a good "CRUD" application and I take my hat off to you.

Defense sector is full of hard problems. A PhD helps, but even with a BS you can be working on very challenging things. When I worked at a defense startup we were designing protocols, writing VHDL and drivers for logic implemented in FPGA's, doing formal analysis of algorithms, doing data visualization, etc. Different people were doing different things, obviously, but even the junior folks were tackling problems that required reaching into their CS toolbox.

Its not an industry that has a great reputation, but what it does have is phenomenal engineers working on complex multidisciplinary problems.

I would like to strongly discourage anyone from working in the so-called "defense sector", which should be (at least in the US) more properly called the "war sector".

Working to enable the more efficient murder of ever more people is highly unethical. Unfortunately, there are plenty of unethical people willing to do this bloody work. But hopefully you are not one of them.

We're living in the Pax Americana, the most peaceful time in human history, enabled in no small part by total American military technological superiority.

Not to mention we're posting all this over a network created for the American war machine, using protocols created by a company that is a key part of the military industrial complex (BBN), which was founded by people from a university (MIT) that was then and is now one of the largest non-profit defense contractors in the country.

Have you been in a "CRUD" job for at least a year? I can assure you, what often seems like a "CRUD" will teach you a lot about architecture, design, requirements analysis, etc. It's actually a good way to teach yourself skills that cannot be obtain via university schooling. Spend a year mastering a framework or two (in a paid project), face some support issues, etc. When the job is boring and you complete your tasks very well, ask for a raise and/or start looking for a new job.

1: the "99%" you're looking at are jobs on a specific niche that uses technology to bounce information from one place to another and display them, mostly known as IT jobs.

2: Hard problems are everywhere. Literally. Even in those "99%" jobs you mentioned -- "doing" such software may not be full of mysteries, but make it evolve without break and respond to constraints are examples of real hard challenges. It's a matter of seeing them and putting it on perspective.

3: there are many ways of framing a problem, making it hard or easier. The harder you frame it, the more instruments you will be required to have to tackle it. Those may involve research skills and fluency in an array of fields that would help understand the phenomena. It mostly depend on where in the spectrum of the problem you want to work, and that's up to you to figure it out.

4: advice to avoid a career working on crud apps: interview the interviewer and find out if the problems of the hiring company are the ones that you value.

5: questions like "where to work on hard/fun stuff" are very hard to give satisfying answers. I suggest you take the time to survey problems you are interested, try talking to people working on them...many problems are tackled by teams, often with people with different background and specialties dealing with different aspects of the problem.

I used to feel this way too. My first job was writing a web app to replace a paper approval/overview process at an engineering firm. Aside from some email notifications, it was entirely CRUD and, to cap it off, I was forced to use classic ASP (in 2005 - asp .net 2.0 had come out by then, making classic ASP 3 generations old).

After 6 months of adding more and more approval flows, I had enough of the tedium, so I started writing libraries to make classic ASP easier to use, including a hash table library. Then I focused on how to write CRUD apps faster, with less code. Then I focused on making it more interactive and wrote an AJAX library. And if you want real difficulty, try making a classic ASP rest endpoint running on decade old hardware over a horribly designed MsSQL database performant.

I guess the point of that story is anything that bugs you on a daily basis can become an interesting and difficult problem to solve. Constraints make simple things more difficult, but you did say the harder the better. :)

If you want to work on hard problems, strive not to have to solve the same problem twice, to try a new approach or a new tool every time you get a "CRUD" task (however you define it). You'll find that if you continuously push yourself, you'll keep growing and will never feel that the task at hand is boring or too familiar. You'll need to take risks, but without risk there's no reward. Regard every programming task, however simple, as a chance to learn something new.

Corollary: if you find that your job doesn't let you learn (e.g. your boss just wants you to churn out as many web apps in as little time as possible), leave immediately (or better, try talking to the boss first :) ).

However, as ef4 and others have pointed out, the real question for you might be "what are the problems I'd want to spend my life on?" Not all important problems are hard - some may even seem too unsexy to be worth working on, but the impact can be huge.

Unfortunately just learning ruby, rails or any other programming languages doesn't prepare you at all for type of challenges you are talking about. I've been working on search, machine learning, prediction models, computer vision and so on and these are absolute cream of the challenging problems out there that I highly enjoy. Other my favorite areas are compilers, kernels, audio and image processing but I'm not actively working in it currently. But to even understand current state of art in these areas, let alone contribute to it, you will need solid foundation build on math (especially linear algebra, probability & stats, calculus), machine learning, distributed computing, graph algorithms, probabilistic data strutures and other CS areas. It takes years to build this theory foundation and perhaps more if you are not already enrolled in to any degree programming. Programming is actually the most trivial part of working in these areas.

Employers who do have these kind of challenging work are never looking for just Ruby or Java guy who knows all syntax, framework and how to code. They would need your experience and academic background to convince them that you do have necessary math and CS foundation to tackle their challenges. The best way to start out for "plain vanilla" programmers is to get in to open competitions like at http://www.kaggle.com/competitions. Also keep an eye out for any companies sponsoring such competitions like Netflix did years ago. The reason this is good way to start is because here would typically be discussion groups around this where experts discuss their approaches for real problem and real solutions and you can learn a lot from this. Once you select such a challenge to work on, give it all to it. Devour every piece of research, CS and math that comes in your way to understand what's going on. If you keep at it for 2-3 years, you would be in might be in good position to demonstrate future employers your work and worth to work on these challenging problems.

try Google, if you are really good and have passion to develop tough stuff like machine learning languages, dealing with large data. By the way have you tried any problem given in http://www.topcoder.com/, if you like that stuff and also can do, Google is worth trying.

It depends on what kind of "hard" you're looking for. I've worked on games (Crash Bandicoot), search (ITA Software), and email (http://inky.com) and all have been hard, though in different ways.

Games are hard in that you're always trying to squeeze more into severely constrained resources, so you end up having to produce convincing but fake approximations of algorithms that would be far too expensive to run in their "real" forms.

Travel search was algorithmically very hard (see the few papers by de Marcken about this for a glimpse into this world).

Email is hard because it combines many disparate skills (design, UX, back-end scaling, protocol details) and because the MVP is incredibly complex.

All involved tremendous amounts of slogging.

Writing a distributed database is pretty challenging. That's part of the reason I started a NoSQL company some time ago. Unfortunately it failed due to lack of funding, but if you're interested, here's a technical write-up:

http://arxiv.org/abs/1302.3860

I can think of a lot of scientific "big data" applications that would be very challenging problems. There are many domains that need to create & automate better systems that Collect > Organize > Parse > Visualize data. Weather, climate, ecology/species, traffic systems, etc. To do these right would be very challenging and would have a big impact on the world.

Now some self-promotion: I'm looking for a full-time partner to help me with Atlastory, an app that visualizes all of history on a map. This fits with the idea above. It's a simple concept, but there's huge engineering & design challenges involved: imagine all the geographic data involved in Google Maps, now multiply that by 1,000, and figure out the easiest way to obtain/retrieve/display that data.

This is a good question. I've worked in a lot of different domains and can tell you that:

1. They all have boring parts

2. They all have interesting parts.

3. Doing the interesting parts well can be very challenging.

CRUD is only boring because of the generally repetitious nature of talking to the database, but repetition usually means that it can be generated. Whenever possible, I write code generators for that sort of thing. This is not only more interesting and gives me time to solve the more interesting problems but also allows me to produce more and better code than I could just typing it all out.

You could get a PhD, I suppose, but hard problems generally require the insight to see them and the willingness and ability to work on them. Sometimes ability requires a PhD, but that's not a given.

Try systems companies (e.g. storage, networking) - you get to build products that leverage core CS concepts - algorithms, distributed systems etc. You also get a chance to use your investigative skills to debug problems like race conditions, memory leaks etc.

I'm running into the same problem. I am tired of CRUD apps, I just don't see myself doing anything new or exciting with them . Last year I was approved in one of the top tech Universities in Brazil and started learning C and C++ and this year I'll engage in programming competitions.

By the way, I've already failed 2 math disciplines in my University and I'm doing them again. But I'm good at programming and I'm becoming way better at Math. Someday after I graduate, I'll try to work for Google or Facebook, do you know if they'll turn me down for the fact that I have failed some math classes (even if I later was approved on them)?

Thanks everyone.

I have a hard and interesting job. I do navigation & autonomous control systems for large mining vehicles. A couple of notes...

1. You want to do R&D. Developing new things is fun; maintaining existing things is boring. However, "getting in" to R&D positions is difficult without #2:

2. Domain knowledge. We almost never hire based on programming skill alone. Programming is a prerequisite, but specific expertise (guidance, navigation & control systems in my case) is the real value you need to bring. Graduate degrees are one way (but not the only way) to move in this direction.

Here are a couple of CS-y but not too math-y suggestions.

Consumer-facing interactive native apps are typically low on CRUD and some of them require non-obvious solutions. You could try to get a job (or just create a product on your own) on mobile or in games.

Another option is working on tools instead of apps. If you manage to get some non-trivial gcc/clang/kernel patches accepted you can try to find a company that works on those.

(I've worked on Google Maps for Mobile and am now building a robotic waste recycler - no CRUD for me since 2002 unless I've decided to build some for my own needs).

I got lucky and transitioned (in the same company) to a search project from CRUD - which I find vastly more interesting.

But my advice is to pick hard problems yourself and try to solve them. You need to start somewhere! Sure its expensive to get your hands on driving car hardware, but there are plenty of hard problems out there that need solving and can be done using opensource and your personal computer.

If you are able to get yourself through a really tough problem, you will come out better on the other side - even if it takes you years to solve.

One place to find hard open problems is at the end of technical papers (e.g. in math, science or engineering). Many of them will have a section talking about issues, future work, ideas remaining to explore, etc.

There are infinitely many hard problems out there. If you want to stay motivated, work on stuff that matters.

http://radar.oreilly.com/2009/01/work-on-stuff-that-matters-...

I don't know how you feel about working for the military-industrial-tech-surveillance-state but companies or universities that get DARPA or IARPA money work on stuff like flying and/or self driving cars, autonomous systems etc (Lockheed, Honeywell, Carnegie-Mellon, Raytheon, for examples). Of course these things will ultimately be used to crush humanity's soul, but they're interesting software and technical challenges.

I work in this industry, and have spent a number of years in R&D. In my experience, the vast majority of software R&D projects are not weapons/military related, and include things such as cyber security, healthcare, big data, cloud computing, etc. A lot of it is aimed at developing innovative approaches to reduce costs in various government/military processes both through automation and doing things more smartly and efficiently.

Sorry, but I don't think there's a distinction to be made, other than one of degree unless the implication is that improving government/military processes is not aligned with the long term goals of the military.

Good point, but I think the average person who has reservations about some military functions isn't necessarily wholesale opposed to all functions of the military.

I was in this business for a while too and did some really cool stuff. But in the end this system transfers public money to enrich these companies; and the DOD basically pays these companies to learn how to develop technology that the company can then make huge profits on. It's a racket. I'm torn about it; philosophically, morally, politically I'm opposed to war and using technology to control people. But the skills I got and the smart people in the industry were great. Plus the projects are cool from like a 13 year old boy perspective, but if you stop to think that big data, autonomous vehicles etc will be used for death and increasing the powerful people's ability to control us, it's might not be worth it.

This sounds like a variant of the meme "I could build stackoverflow in a weekend".

If you think you can build 99% of the apps out there, go out and try to build a moderately complex app. Make it better. You'll see there are all types of challenges from the technical, to the social to the political in small and large companies. Being a software developer is about solving all sorts of challenges, and not all of them are technical.

I don't think thats its a variant of the meme "I could build stackoverflow in a weekend." It doesn't have the in a weekend part.

The OP seems to consider getting a PhD an acceptable route. He's asking for directions, not a shortcut.

Twitter, Facebook, Stackoverflow are all really just CRUD apps. They face social and scaling challenges everyday among other technical challenges. I don't think he's giving CRUD apps enough credit. There's also security challenges, etc.

Pretty much every single company out there has boring programming work to do that would be redundant if they could solve some harder problems (which they may or may not be aware of). Competent programmers will notice the bigger picture and attempt to solve the hard problems. If you want to be hired specifically for hard problems, try academia or some research insititute / R&D department.

To be blunt, the simple problems are hard.

Concurrency. Locking. Cache invalidation. CAP Theorem for determining truth in distributed datasets. Feel free to solve any of them.

For the record, "I could do it if I had the time". Do it. It'll be awesome.

Reinventing the wheel is a great excercise, as long as you accept that your wheel will probably be inferior to the existing one. Or maybe it won't and it'll be the next big thing.

TL;DR, go write some code.

relevant HN thread: http://news.ycombinator.com/item?id=4968452

Interdisciplinary projects have a lot of potential to be challenging and rewarding. Bioinformatics, Computational Neuroscience, etc. - these are your phd jobs. Another cross disciplinary project is entrepreneurship (you have to do everything - marketing, finance, product development, etc.) and this doesn't require a phd.

Certainly I'm biased, but I think working at a place like http://istrategylabs.com is a place to do this.

We have to invent new things everyday, and due to our focus on hacking the digital and physical worlds there's so many possibilities.

We work on hard stuff (http://vicarious.com/) and we're always looking for exceptional people

Get into digital signal processing / digital communications. They are awesome fields that are challenging and rewarding.

What is your topcoder ranking?

You do realize that CRUD hasn't even really been solved yet, right?

Get a PhD, or get a high paying job and do interesting stuff of weekends, or do your "self-directed" work as a pauper with option to do a startup or open source project that gets you a job later.

Get a job at a big web company that has distributed systems.

But one day you will learn that even "exciting/Challenging" work is 99% perspiration.

I happen to enjoy easy jobs. It allows me to use all of my mental energy toward my own startup :-)