My Cities Experiment - The Results
(BS) Below is the report I received from Brendan McKay with the results of the experiment I asked for and some additional ones that he did. My comments are interspersed in italics and surrounded by my initials. (BS)
Here are the results of the experiment you commissioned.
(BS) One problem I didn't foresee is that there is always some wiggle room. The students indicated that it was unclear whether a few city references were to cities in which a Rabbi lived - for example, a reference to where his teacher lived. (BS)
The three students marked 6 city names as unclear, so I calculated the results with the unclear cities all included, and the results with the unclear cities all excluded. All ranks are out of 1 million random permutations, using a program equivalent to the els1.c program of Yoav Rosenberg.
(BS) The WRR method that gives 16 out of a 1,000,000 (i.e. 1 out of 62,5000) takes the smallest of the P1-P4 permutation ranks (4 in their case) and multiplies by 4, so for example in the first list we get 4 times 162,471 out of 1,000,000, i.e. worse than 1 in 2 (BS)
The data received from the students was used exactly, even in the case of a few errors that we noticed.
Unclear cities included:
Unclear cities excluded:
In addition to the data collection by the three students, data was collected by two experienced older persons. In contrast to the above, all known errors have been corrected. As well as correction of errors, there were some differences of opinion as to whether cities satisfied the rules for inclusion.
Data was collected separately for the Margaliot Encyclopedia and for Encyclopedia Hebraica.
Margaliot encyclopedia - Unclear cities included:
Unclear cities excluded:
Encyclopedia Hebraica - Unclear cities included:
Unclear cities excluded:
Both encyclopedias combined and duplicates removed - Unclear cities included:
Unclear cities excluded:
(BS) My experiment asked for ALL cities in which the Rabbis lived; Gans' experiment only cities of birth and death. In addition, McKay knows that Gans allowed two prefixes since McKay has seen a pre-preprint that has not been publicly released. (BS)
An additional experiment was performed to make clear the comparison with the previous Inbal-Gans experiment. In that case only cities of birth or death were used. In addition to the raw city names, the prefixes QHL and QHLT (community of) were uniformly added, and only P4 was used.
A very strong result was claimed.
(BS) The data collectors were asked to categorize the cities they found into groupings such as "birth", "studied in", etc. (BS)
We have done the same thing with the cities allocated to the categories "birth" and "death". Here are the results for P4, first with the unclear cities included, then with them excluded.
Margaliot, as collected by the yeshiva students:
Margaliot, as collected by the older persons:
Hebraica, as collected by the older persons:
Combination of the last two sources:
(BS) The best result here is less significant than 1 in 10. Thus the better result claimed by Gans is clearly entirely due to the use of arbitrary spelling rules and this experiment is beset by all the same difficulties as the original WRR experiment. (BS)