A data transfer tool.

So many people are working with data professionally, and there are hundreds of solutions for analytics, data warehouses, time series databases, etc.

Yet getting data from one silo to another is annoying, and usually requires writing complex scripts to transform data from one format to another. If you're not familiar with something like Python that will be very difficult.

I'm thinking of things like this:

- download data from a REST API and store in a CSV file

- scan log files and put data into Redshift

- copy data from MS Access to PostgreSQL

- extract info from email messages and store it in a spreadsheet

There are solutions to some of these problems, but I think there is a huge opportunity in this space. Right now a lot of data science still requires a lot of irrelevant technical knowledge, and a powerful GUI could allow people to focus more on what they want to do rather than how to do it.

Great ideas. A lot of the freelance Python coding work I do is data transfer either from database to database, file to database, API to database, or some combination thereof.

Can you give an example of file to database and API to database? As long as the file is a CSV, there are tons of tools already to import them into a DB, isn't it? Are you talking about unstructured files here?

Generally it looks like: unstructured mess -> CSV -> database, or mess -> structured intermediate format -> whatever. Or database -> nicely formatted report that makes CEO look good. Even structured (weird XML) -> sort of structured (XHTML).

There was no generic solution although I've ended up reusing a lot of code.

One of the first gigs I got was to process about 7,000 pages from a static website, scrape out the relevant data, and plonk it in a database so they could run queries on it. :)

You could probably latch onto a common use case and build a solution for that e.g. Blogger to Wordpress transfer - something I've had to do in the past by building some custom tools. (That was years ago though - I've not looked at current state of play with regards tool availability). Not suggesting that would be your use case though - just an example.

One thing I do is import data from Github issues into a table. There is no CSV export, but there is an API. So I first write a script that lists issues. They are paginated, so I need to keep making requests until I have all of them.

Once I have a list of all the issues, I need to make one request per issue to get further details that aren't included in the original list.

Then I need to extract data from all the downloaded JSON files and combine it into a table.

It's conceptually not a hard problem, but it takes a lot of trial and error to do that if you aren't very familiar with Python.

I wish there was a nice GUI where I could just click a few buttons to do that. Designing that GUI is probably not easy...

Woah, I wouldn't have guessed. I find it crazy that Github doesn't have a download all option, either in the UI or at least in the API.

Any more examples you can share?

My online store has an API endpoint to get order details, but they don't have a way to list orders. So to get sales numbers for last week, I had to write a script that uses IMAP to log into my email account, checks the folder with order confirmation messages, parses the order numbers from the email subjects, then makes an API call for each order number to get order details. Then I get foreign exchange rates for every day of last week from a different API, put it all in a PostgreSQL database, and then I have a view that estimates weekly revenue...

What platform does your store use?

Thanks for replying, after reading your comments, I did a quick search. There are tools like Panoply, Blendo etc, but I couldn't find many with the level of customization that you described here.

I was talking about Fastspring, they run my store, and they offer the API and send order confirmation emails. The IMAP thing is just what I use to get a list of orders.

(They also support webhooks for incoming orders, but that would require me to run a webservice... the IMAP hack seemed easier)

Edit: I looked at Blendo, it sounds like that's kindof what I'm looking for, but it's probably cloud based and they don't even have pricing on their website ... not exactly the lightweight tool I had in mind.

If you don't have an idea do not take off for a year to "play house" with a startup. If you've read any of the advice on this site the startup it's mostly about your idea. And it's best to be passionate about that idea and have a strong connection and urge to solve it.

I can't imagine anyone being successful with someone else's idea.

If you want to take a year off for fun or any other reason that's fine of course.

I posted this on another thread, about ideas I'd like to work on:

1. Uber for private tutoring/students. I live in a university town and there's plenty of college kids wanting to teach schoolkids.

2. Random story/character generator based off tropes.

3. Punch card for babysitting, especially for the late night tiers that babysitters are reluctant to charge extra on.

4. Recipe app, focused on instant things like bread makers and pressure cookers.

5. Github but for recipes (this is really just an excuse to make fork puns)

6. A chat with anonymous strangers community, similar to Omegle, except you post something similar to a tweet, and people can chat with you based on it. So you could make a post complaining about your boss, or how happy you are to get a job, then someone can chat with you about it. My main worry is that this could degrade into 4chan and it would be an uphill battle to moderate it.

7. Gamification productivity app. Probably just a checklist, ala Habitica, or it could be integrated with Pomodoro Technique.

Number 6 is basically the Secret app, though the community quality there isn’t great.

Also, googling "github for recipes" brings up a number of options as well.

> Looking for ideas...

Look for problems to solve. Problems are goldmines.

On this subject, Peter Diamandis is brilliant> http://podcast.diamandis.com/2015/10/19/episode-11-problems-...

Probably better to work on one of your own ideas. Write down every idea you have, write out pros and cons for each, can you combine them? Then figure out simple business plans for each, keep it simple, find you competitors and learn from them, find your customers and learn from them, what resources you need to build a mvp, how are you going to market that, how do you plan to grow the business from the mvp.

From all this research hopefully you will get a better feeling for which of your ideas are worth following up, and their potential.

Treat your business as if you were investing. Would you spend your hard earned $100k on a startup with no idea, no plan, and no exit strategy. And would you bank all of your life’s savings on it?

The time to quit and do this is when you have some customers and revenue, a plan, and need the time to grow it. Take a vacation, start it, and get your first customer, and work on it in your free time.

Good advice. I would almost be tempted to invest the 100k or whatever in long term index-linked investments and work on the side project on the weekends...

If you want to take a year off for a break more to you. But I wouldn't quit to work on an idea that you don't have yet. You'll be wasting a precious opportunity that can be taken advantage of when you do have something you're passionate about to work on.

There is a thing in my GitHub (in profile) that you could work on. It's possibly not up your alley as it's probably not monetizable. You've implied that matters to you, but not explicitly stated it. So I'm tossing that out there.

From 22 hours ago: https://news.ycombinator.com/item?id=19164037

>I’m taking a year off to live off my savings and work on an idea, but I don’t have one

Why take a year if you are clueless as of what you want to do? Do some traveling and air your head. If you still cannot find something you want to do, go back to your 9-5 job.