I would then take them into my office and hold an inquest on how two completely different data-sets got mixed up in the first place.

At which point I would probably find out Alison and Charlie have been knocking boots on company time.. Alison and Charlie would then be fired as this is against company policy.

Due to the difficulty in finding new jobs Alison and Charlie's fledgling romance would end... leaving Alison unexpectedly pregnant and Charlie with an 18 year bill for child support.

The morale of this story? Label your fking work.

Note: I was unable to solve this problem as my math skills are poor.

It's in scipy.stats.ks_2samp

Results:

Sets| D |p-value

-----------------

A,C |0.275|0.080|

B,C |0.175|0.531|

-----------------

A,D |0.125|0.893|

B,D |0.275|0.080|

-----------------

A,E |0.100|0.983|

B,E |0.300|0.043|

-----------------

A,F |0.300|0.043|

B,F |0.100|0.983|

-----------------

As far as the test goes, if D is small and p is high, you cannot reject the hypothesis that the two datasets came from the same distribution. The p-value is roughly how often, randomly you would get similar looking data assuming the null hypothesis (in this case that they are drawn from the same dataset)

In light of this evidence, if they are not lying to us, and really each of these sets came from and A-like or B-like distribution, I'd say fairly confidently that:

F is B-like

E is A-like

D is A-like and

C is B-like (though with lower confidence)

The box-plot: http://i.imgur.com/epPw7.png seems to confirm.

Actually, just looking at the variance of the columns tells the same story as you've discovered above:

> var(data) A B C D E F 117.19610 20.49239 33.13114 90.62195 115.39044 27.34298

A, D, E are in the same group, B, C, F are in the same group.

there is a 98.3% chance that f is b-like there is a 98.3% chance that e is a-like there is a 89.3% chance that d is a-like there is a 53.1% chance that c is b-like

If these are multiplied together, it appears that there is only a 45.8% chance that they are all classified correctly?

The main thing is, you know that C, D, E, and F came from either A or B. The p-values above don't account for that; they just say what's the chance, due to random fluctuation, that a sample could have come from from the same source as A.

That's reflected in the fact that the pairs of p-values don't add to one! (Like (A,C) and (B,C) in the table above.)

You also implicitly know that at least one of {C,D,E,F} is A-like and one is B-like (otherwise there would not be a problem). So even if you know P(X and Y have same source) for all (X,Y), which you don't, you couldn't multiply them.

Finally, the p-value returned by the KS test will underestimate the true probability of discrepancy. This is because it's only looking at one thing, the max value of a CDF difference. The significant differences between the distributions may lie elsewhere, like in the tails, and the KS test is known to be relatively insensitive to tail behavior. (Although at n=40 you won't be able to see far into the tails.)

There are a host of other tests that use the same idea (empirical CDF difference) but weight differently. Some can be more effective than the KS test if you're looking for certain types of difference. Here's an OK overview, albeit for the goal of assessing normality:

http://www.instatmy.org.my/downloads/e-jurnal%202/3.pdf

In a real problem, it's always a good idea to use more empirical-cdf tests than just the KS test, to compare variances and other moments as some people in the thread have done, and to make histogram or CDF plots -- especially if you're in just 1 dimension and the plots are easy to interpret.

Edit: If you want to try, copy the data and click the "Paste data from Excel link" on the left under the Data Tab, paste and load. Then choose histogram under the Graph tab. It plots the X var (ignores Y). Sorry it's a bit shabby, I haven't worked on it in a while.

Anyway, a serious approach to this problem would require comparing against solid real-life data. Off the bat assumptions about distributions of real life data are often very wrong.

So D and E are temperature, and C and F are weights.

Orange http://orange.biolab.si/ "Open source data visualization and analysis for novice and experts. Data mining through visual programming or Python scripting. Components for machine learning. Extensions for bioinformatics and text mining. Packed with features for data analytics."

Weka http://www.cs.waikato.ac.nz/ml/weka/ "Weka is a collection of machine learning algorithms for data mining tasks. The algorithms can either be applied directly to a dataset or called from your own Java code. Weka contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization. It is also well-suited for developing new machine learning schemes."

for i in {1..6}; do COL='$'$i; awk -F, "{delta = $COL - avg; avg += delta / NR; mean2 += delta * ($COL - avg); } END { print sqrt(mean2 / NR); }" list.csv; done

[troll]

* By hand?

* With Excel?

* With R?

* With processing?

I'm a little sad to see that this item has been flagged heavily, but I guess there are people who think this is sufficiently off-topic that it should be in the same category as spam.

#copy data into a txt file without the first two lines from the xls

par(mfrow = c(3,2))

dat <- read.table("data.txt", sep="\t", header = TRUE)

for(let in c("A","B","C","D","E","F"))

plot(density(dat[,let]), main = let)

easy...(why doesnt HN recognise newlines?)

It treats single newlines as the same paragraph and double newlines as a new one. Alternatively, if you want to post code, you can indent it by four spaces. Like this:

    par(mfrow = c(3,2))
    dat <- read.table("data.txt", sep="\t", header = TRUE)
    for(let in c("A","B","C","D","E","F"))
    plot(density(dat[,let]), main = let)

I offer you knowledge obtained by sheer trial and error. :)

In a more general sense, I would tend to use a Self Organising Map (http://en.wikipedia.org/wiki/Self-organizing_map) to identify the two groups of data, and then discrimante between them. The dimensions of the SOM would be different statistical analyses of the data sets - what is the mean, the absolute spread, the standard deviation, the median, etc.

The nice thing about using a SOM is that it will show you whether or not you have succeeded in finding a measure that can successfully discriminate between different data sets - whilst at the same time actually doing the discrimination for you.

EDIT: I realize that this was a somewhat curious result.

The means are all the same.

The two examples have the same mean and median, but differ substantially in their min/max/standard deviation/median absolute deviation. Pick your spread, they differ. I got this by poking around in iPython.

Find the standard deviation of A. Reshuffle A and B together, taking half of the resulting list, calculate the standard deviation, record it and repeat that a thousand times. See that you basically never get a standard deviation as large as you did for A alone, so they meaningfully differ.

Repeat on each of the data sets, and you see that D and E are plausibly similar to A whereas C and F are not. Repeat for B to see if we're being fucked with, and lo and behold it all seems to work out, though D is a little iffy.

I used numpy and ten lines of glue code. Happy to post if there is interest.

Here's the result in Excel: http://ii.snag.gy/vSt4H.jpg

Alison's lists are A, D and E

Charlie's lists are B, C and F

For the sake of completeness, my first steps before sorting and plotting were to:

  1. take the mean, median and mode of each set,
  2. try and fail to use more sophisticated statistical analysis [1],
  3. have a look at the minimum and maximum value for each set.

[1] In fact, could anyone point me to a refresher? I'm talking about distribution curves, regression, that kind of thing. I seem to have lost hang of it — couldn't pass my own "5 minutes to implement the simple thing" criterion.

Checking it seems that this is borne out. So is the data real? What is being assessed here?

Interesting to see that people prefer std dev to variance for this problem.

http://www.hostmypic.net/pictures/93969c88a3ecaac84ba81c8d38...

And it is almost obvious that we have two data sets: (A, D, E) and (B, C, F)

I am quite rusty on this stuff I must say!

I guess one would correlate each row to the A and B one and look which correlate the most?

http://i.imgur.com/Dg6OT.png

but to be honest the first thing i would do is google for suitable approaches...