I've always found enjoyment in finding statistics in things that interested me, so here's a few thoughts on project ideas. The main thing is to find a data set that interests you, and use that interest for fun or for profit:
- pull down your facebook backup, run it through sentiment analysis, throw it in a db, explore yourself through facebook's eyes!
- a while back I was looking to buy a car and used scrapy to process cars.com, ran their vins through a look up tool to find cars that were actually manuals (and not just manumatics, seems few can tell the difference these days). Found reasonable national prices, average miles, etc.
- interested in politics? pull down the data sets for voting records, explore your local politicians' voting records, compare them to national averages or historical information, etc.
- interested in movies? find movie datasets (or scrape imdb.com, themoviedb.org, etc) and find which genres pay the least per actor, have the smallest cast, etc.
Lots of datasets are available online, if not in machine readable format, then in a format that can be easily scraped. Have fun!
An registration company I used to work at was using .NET 1.1. Being the super ambitious junior developer I was, my first move was to upgrade our software to the latest and greatest: .NET 2.0. After passing all the tests and signed off by QA, we moved it to production, and we pat ourselves on the back, having done A Good Thing (tm).
Soon afterwards, however, we started receiving reports of our users not being able to refund or charge credit cards. All that information should have been in the DB, encrypted! We quickly discovered that, on occasion, the encrypted data getting corrupted. Immediately we did what every engineer would do in our place - blame the previous engineer's code, then try and find the bug that would prove our theory right. After days of studying source code and testing theories, nothing explained occassional corruption.
Eventually we traced the beginning of our problems back to our server/framework upgrade, and found a Backwards Incompatible Change: invalid unicode code points would now be silently dropped, rather than being allowed. It turns out that all of our credit card numbers were being encrypted properly, but then DECODED using the UTF-8 Encoding and stored in an NVARCHAR column in the DB! Everything was fine in .NET 1.1 (and SQL Server 2000) but .NET 2.0 silently drops the invalid UTF-8 code points. With those code points missing, it was impossible to decrypt the data and do anything with it.
... I suppose that makes it more secure though, so there's that ...
We felt that .NET 2.0 was a big enough upgrade that it was worth adding some new warts to our system. The final hack: we found an unused pc and built a .NET 1.1 web service with two functions: encrypt/decrypt. We store credit card numbers in the database in plain text, make a call to this webservice with the row id, and it encrypts the data. This solution lasted almost 5 years before our boss accepted the pain of an hour of down time and we exported/decrypted/encrypted/imported the entire db.
First and foremost, remember that you're writing code to bring in customers. Your codebase can be beautiful, pragmatic, semantic, and have 100% test coverage; if you don't have any customers, you don't have anything.
"Unnecessary complexity to the codebase"
It depends on what you mean by unnecessary. If you mean "won't bring in anymore customers", have that conversation with your managers. Not all of them are brilliant, and no one gets it right 100% of the time. If you can prove that the feature doesn't provide value, have that conversation with them.
On the other hand, if your boss ignores your input, and you're 1000% sure that there are other features that are more valuable to your business than the one in question, you can always push that one to the back and work on something that's more productive to the company. Depending on your political and professional circumstances, your boss may not notice or care, and their boss may forget about their red herring feature; you might be able to side-step the conversation altogether. This will only work if there's more than a few items on your plate that need to get done soon, and this feature can get pushed aside without delaying or blocking anyone else.
Bear in mind that if you go this route, you're putting yourself, your career, and your neck on the line. If it turns out that it wasn't a good idea and everyone agree with you, you'll look like a genius and gain some clout as a clairvoyant; if it turns out it was seriously necessary, you'll look like someone who pouts when they don't get their way. Either way your boss may also hold a grudge. I'm not saying it's the greatest way to go, just adding it as an option. It's helped me more than a few times in my career, but it's also frustrated my bosses a few times. Be gracious if you're shown wrong though, and quick to admit defeat if it's obvious you chose the wrong path, and you should be fine no matter what happens.
Kabbage is a financial technology company that is revolutionizing the way small businesses get capital to grow. We have a patented technology to perform real time analytics to determine the credit worthiness of a small business, and then to provide that business with funding to grow. Small businesses and online merchants are underserved by banks who often view them as just a credit score. Kabbage leverages data sources such as Ebay and Amazon shopping activity, UPS shipping data, Facebook, Twitter, and dozens of other data sources to turn the lengthy and cumbersome 7+ week process of applying for a loan that often results in a “no”, into a seamless process that provides instant funding to businesses in less than 7 minutes. We are passionate about helping small businesses to grow and prosper.
Our systems are build on top of ASP.NET MVC 3, WCF, jQuery, Entity Framework, KnockoutJS, and other delicious code bits.
