The idea of the project is a new approach into collaborating on knowledge, by splitting the knowledge into smaller parts, and then being able to handle them in an easier way.
Most of the definitions come from Wikipedia. We started off trying to have people enter facts manually, but then I figured that we should leverage all the knowledge of Wikipedia that is freely available.
So I wrote a bot that is not particularly clever, but most of the times good enough for fetching facts from Wikipedia.
By the way, it can also extract facts from the German Wikipedia (there is a German version of Factolex as well: http://de.factolex.com/ as my mother tongue is actually German).
Actually, that isn't the wealth creation. It is am important market and business function, but the process of want creation and distribution are actually eliminating market inefficiencies, not increasing wealth. Of course, my argument is based upon the simplistic economic model, and I pay no heed to people that believe want creation yields emotional value through the deliverance of material satisfaction. Now, if you mean to say that business people are responsible for increasing factor income, than yes, exactly.
wget is synchronous while Twisted is an asynchronous networking engine. This means that you don't need to wait for a request to finish before making another one (or making pancakes, or doing whatever you want).
I essentially wrote a parallelized version of scrapy which has the ability to make hundreds of requests per second, depending on available CPUs. You could never achieve that level of performance using wget.
This is great. I was running threads on a current crawl job but the real bottleneck is BeautifulSoup and not the network. So splitting the project into threads(while it helped about 10%) wasn't really necessary and Twisted probably would have done the trick.
Because the method is explained, but not the exact path. There is a process to follow to help you make something people want, but no way of knowing what people want without trying.