I understand the rest of your point, but the underlying assumption is very wrong.

This is the opposite of how useful research and/or data science works. Data should be taken as is and then learned from. It certainly should not be gathered in an effort to directly prove a conclusion that you are already "convinced" of.

It's very disheartening to see this as a comment on a data science article on hacker news... unless you're being sarcastic/ironic? This has to be sarcastic/ironic, right? Right? :(

Get down off the pulpit. If the data said the opposite, I would instantly change my worldview.

But you are correct, in that my wording or attitude was incorrect. I should have said "I suspect" instead of "I am convinced" and instead of "might support this conclusion" I should have said "might support or refute this hypothesis".

Which is to say, don't scientists at least have a hypothesis in mind before they collect data related to it? Otherwise, why would you be testing at all, and for what exactly? You can't just take millions of data points, put them into a blender and get proven theories out of it!

Anyway, that's what I'd have at this point, a hypothesis. I should have used that wording, my bad.

One camp says "collect any data that might be relevant, and then begin looking at the data to try to figure out what the hypotheses should be"

The other camp says "formulate a hypothesis, and then find the data you need to test that hypothesis".

The problem with the latter approach in the social sciences -- or any setting with lots of unknown latent variables -- is that it's often possible to find some data set for which a given hypothesis holds with p < 0.05. So whenever there are a lot of latent variables, it makes a lot more sense to construct a high quality data set first, and then start hypothesis testing.

The problem with the former approach is that you really need to know "this set of data is probably really interesting / representative for an entire range of hypotheses about topic X", but that's often not clear from the outset. And it's often the case that for any particular hypothesis, there are lots of other data sets you might know could also be relevant.

In any case, whenever there are lots of unknown latent variables, cherry-picking data sets that confirm your hypothesis is a really good way to lead yourself astray.

My solution is to just avoid working in fields with lots of latent variables, but that has limitations was well :-)

"Bias is an inclination of temperament or outlook to present or hold a partial perspective, often accompanied by a refusal to consider the possible merits of alternative points of view."

http://en.wikipedia.org/wiki/Bias.

How is the prior negative if it is accurate?

So basically, you're arguing that "common sense" tells you that women at tech conferences are recruiters or HR. And from a common sense perspective, it may be right. But you didn't say common sense. You said "How is the prior negative if it is accurate?"

By definition, the prior is not going to be accurate for a significant minority (if not a majority) of the women at the conference. And every time you're wrong, you are negatively affecting individuals. Don't want to talk to recruiters? You avoid them. You don't bring them into conversations, or don't assume they can keep up. Your avoidance harms their networking opportunities. You're hurting them.

This, this is why bias matters.

The thing is, this sort of thing is pretty easy to manage in real life - just don't make assumptions, and ask people about themselves. This also falls right in line with the classic advice from "How to Win Friends and Influence People". People like you more because you're interested in them, and you don't make harmful assumptions about them.

Sadly, too many people think "Well, I'm not sexist/racist/homophobic", and make excuses for continuing their pattern of bias rather than really questioning their own behavior and finding better ways to act.

Assumptions are useful, which is why we use them. Yeah you should ask people about who they are and what they do, but do you really want to spend 15 minutes talking with a recruiter that you could have spent talking with a programmer? No, so you have to avoid the recruiters (lets you be stuck with them) and the best you can do is work based on your assumptions.

If you don't like it, try to replace the word assumptions with Bayesian weighted probability.

Secondly, suppose the prior is accurate. Lets take a very simple model, suppose g_woman = 0.25 and g_man = 0.05. Further, suppose networking with a developer has a value of 1 utilon and networking with a recruiter has 0 utilons of value.

If I network in order to maximize utility, based solely on my prior (i.e. ignoring any posterior info), I've added 95 utilons to the world for every 100 people I network with. If I behave irrationally and network with men and women equally, I've added only 85 utilons to the world. I've harmed 47.5 men in order to benefit 37.5 women - on net I've harmed 10 people.

(If posterior information is available, then you can even increase utility beyond 95/100.)

This is why math matters, and why carefully thinking things through rather than spouting incorrect soundbites (as the author does) is important.

Also, the particular model I use only requires a mean positive utility - even if we take a model like yours, the conclusion is unchanged. Variance simply goes up, but the best option is still not talking to women.

I mean, if you only talk to men and then measure where you derived utility, I'm not sure you've properly evaluated the model yet (if you interact with a certain percentage of women at conferences and keep track of all this in order to make sure that your model is working out properly, well then, more power to you).

And if I made this point in a non-political context - e.g., "you must account for the base rate for $disease when interpreting a $disease test" - no one would be disputing it.

If you or anyone else here can even present a (utilitarian) model where ignoring base rates leads to a better decision, by all means do it. I'd love to see this, though I suspect the actual outcome of such an effort will merely be the person attempting to do it gaining a much better understanding of Bayes rule.

Also, I now strongly recommend you go through the exercise I suggested. Then you'll realize that accurate tests do NOT somehow eliminate the need for models or accounting for base rates.

I'm reaching for the opposite point. Some sort of meaningful evaluation of actual outcomes is necessary to make utilitarian decisions. If the evaluation of the outcome is arbitrary, then so is the decision.

Second, network effects matter. You're not just creating utilons for yourself, you're creating a system that creates utilons. Even within the small and inaccurate world of your invented model, you're not following through to conclusions.

Third, you're making some very arbitrary assumptions, and ignoring other reasonable assumptions. For starters, do you give everyone you network with equal time? If you encounter a recruiter, do you give them the same amount of effort you would give to an engineer, or do you extract yourself and move on to the next person? The expense of equal input is not nearly as high as you're presenting here, assuming you don't "behave irrationally" and give everyone equal time whether it's effective or not.

Pretending that bias and excuses are intellectual rigor by inserting arbitrary, invented numbers into an imaginary equation is just an appeal to authority fallacy.

Secondly, I didn't make any assumption that the utility is all mine. The 1 utilon can be split between both parties in some arbitrary manner, it doesn't change the result.

Third, you are correct that I my constraint may not be #recruiters + #developers = 100. It might be alpha x #recruiters + #developers = 100 for alpha < 1. That doesn't change the optimal course of action - my best bet is always to minimize time I spend with recruiters.

Now if you think my model doesn't work, present a better one. But if you are making a fundamentally moral and non-utilitarian point ("networking with lady developers is intrinsically good no matter how many puppies get killed!!!!!"), make that point and don't waste time on positive claims if the truth of the positive claims is irrelevant anyway.

Also, you seem to wildly misunderstand what an appeal to authority is. An appeal to authority would be "I asked Eliezer Yudkowsky and he said I was right." Writing down a simple mathematical model is not remotely an appeal to authority, that's just careful reasoning.

Since I presented every step of my argument, and you examined it, it is by definition not an argument by authority. It's simply an argument.

If you want to make a rigorous utilitarian case, do it. Simply pointing out some (non-)problems with the model I presented is not the same thing. All you are doing is arguing that there is more uncertainty than I believed, and then making an unjustified assumption that the uncertainty somehow supports your case.

Also, I didn't "ignore" your moral argument. I specifically asked you to make it - "What morality do you take?"

Beyond that, I do question utilitarian morality, for exactly these reasons. If it were software, it'd be a code smell. It's very easy to turn into justification for all sorts of foul things, and the track record of utilitarian morality is very ugly - like millions of dead ugly. It sure sounds good, especially if you're smart and used to being right on logical issues that don't involve squishy emotions. But it's a dangerous path.

I also have no idea where you get "millions of dead ugly" applied to utilitarianism. I also don't get why you think "squishy emotions" like tribalism, desire to affiliate with high status people, or envy of others will somehow save us.

Maybe if you actually wrote down a model I'd be able to understand what you are trying to claim.

I've already alluded to problems with your model - for example, the idea that equal resources are given to good and bad connections. Another problem is the idea that all benefit equally from connections, when it's obvious that the resource-starved benefit more than the resource-rich. Your model is deliberately starving valuable connections (female engineers). It's also reducing the quality of interactions, adding unnecessary noise to the system in the form of distrust. There are many, many shortcomings with it.

Now, you could just try to keep adjusting your theory to conform to reality, making it increasingly elaborate. Or, you could do the engineer's approach and find a solution that works far better, even if it isn't as pretty.

I already showed a simple extension incorporating this that yields the same result. You are still best served by focusing your attentions on folks who are most likely to not be a recruiter, the delta will simply be lower.

If you want to apply some diminishing marginal returns theory to connections, you'll get a concave optimization problem. You'll still need to multiply the derivatives by 1-P(recruiter), skewing your networking efforts towards men (though not 100% towards men anymore).

The model is, indeed, deliberately starving anyone deemed more likely to be a recruiter. That's the whole point.

Or, you could do the engineer's approach and find a solution that works far better...

You have not even begun to show this.

I strongly suggest you try to cook up any toy model where ignoring base rates helps. I'm dead certain you'll fail (there are theorems), but you'll learn a bit about decision theory in the process.

This improves the position of everybody in the network: those who want to talk with recruiters can do so, recruiters don't have to talk with anybody who would only waste their time and there be no reason to assume that women developers are not since gender would be a strictly inferior signal to the (non-discriminary) badge color.