September 18, 2020

Magic pencils

Writing about web page

Statisticians learn to smile politely when people at parties roll out the so-called Mark Twain / Disraeli quote.

But the subject of statistics is like the magic pencils described in this charming story (less than 2.5 minutes); correct use will lead you to The Truth … .

The story is at the beginning of a full lecture on Youtube, and corresponds to Marie-Claire van Leunen's chapter 35 of Knuth, Donald E, Tracy L Larrabee, and Paul M Roberts. Mathematical Writing. Washington, D.C.: Mathematical Association of America, 1989.

July 03, 2020

Stats MSc student publishes in Early Medieval History

Follow-up to Perches, Post–holes and Grids from Random Curiosities

PEMLIt is not unheard-of for Stats MSc students to find their MSc dissertation leading to a publication, but rather less common for the publication to be a component of a book on early medieval history! But that's what happened to Clair Barnes' MSc dissertation ("Statistics in Anglo-Saxon Archaeology", Department of Statistics, Warwick, 2015); you can read all about it in:

Barnes, C., and W.S. Kendall. “Perches, Post-Holes and Grids.” In Planning in the Early Medieval English Landscape, edited by Blair, Rippon & Smart, Liverpool University Press, Appendix A, 213–31, 2020.

Clair started off studying English Literature as an undergraduate at UCL, but then took an OU degree in Math & Stats while working after graduation. That led to a Warwick MSc in Stats and most recently to a return to UCL, working for a PhD in statistical meteorology at UCL. Statistical science leads to all sorts of unexpected adventures ...

March 17, 2020


Writing about web page

My daughter-in-law Michelle is part of a team that has just published something rather interesting on controlling the epidemic. #VeryProudIndeed

December 17, 2018

Perches, Post–holes and Grids

Writing about web page

I and my MSc student of two or three years back, Clair Barnes, produced an appendix, Perches, Post-holes and Grids, for a book being prepared by John Blair et al., arising from the project, Planning in the Early Medieval Landscape. You can find it on arXiv <> of course! The appendix is aimed at demonstrating the application of statistical methods to the analysis of archeaological data, typically expressed in graphical form, with the objective of assessing the extent to which the spatial configuration exhibits planning by the original architects. Typical questions: did the builders use a common unit of measurement over a wide geographical region? to what extent is there evidence that they used a grid pattern when designing groups of buildings?

The name of the game is to contribute a statistical assessment to be mixed in with all sorts of other historical evidence. It's fun doing statistics in new areas like this: one learns a lot of stuff one didn't know before, and it provides a brilliant excuse to visit Anglo-Saxon Kingdoms <> during the working week.

Crowd–sourcing data

Some really good ideas being implemented recently about and around the idea of crowdsourced data, For example:

Of course the cartoonists got there first:

Crowd-sourced data

Noise to Signal: Rob Cottingham

Gold Access and so on

Writing about web page

Lots gets said about the importance of open-access publishing. Researchers are under pressure to publish papers which are "Gold Access" (translation: they pay the publisher quite a lot of money so that the paper can be accessed freely by all and sundry). Many people discussing this, and/or making policy decisions, appear not to have noticed that in many research fields new work is invariably released as a freely available preprint using the wonderful arXiv <>, for which the publication cost is extremely low (mostly met by academic institutions). For example virtually all of my work of the last 14 years can be found there using <>.

The web-comic xkcd makes the point well <>.

October 16, 2017

Password strength

Writing about web page

Today in the ST116 group we discussed how to build strong passwords. A good exposition can be found in the xkcd cartoon referenced as the weblink for this article.

October 08, 2017

The media has a problem with uncertainty

Writing about web page

Not just the media, but it's a fair point. Have a look at what Nate Silver has to say.

September 12, 2017

The secret of success?

This TED talk makes a lot of sense to me, and chimes in with more than 35 years of lecturing experience. Have a look and see what it suggests to you.

Secret–sharing and independence

The following remarkable procedure is entirely feasible: if I have a class of n students then I can distribute n different binary images, one to each student, such that each student's image looks like white noise,

typical image distributed to a student: just white noise!

and yet if all images are combined together in the right way then a meaningful picture emerges.

result of combining all images using multiplication (if black pixels are coded as -1 and white pixels are coded as -1)

What's more, I can arrange matters such that if any strict subset of the n students tried to collaborate, then all they would get would be more white noise, no matter how they manipulated their n-1 images!

So any strict subset of the students would possess no information at all about the butterfly picture, but combined all together they would be in a position to produce a perfect reproduction of the image.

How can this be done?

  1. Code each black pixel of each image as -1, each white pixel as +1, and view each distributed image as a long vector sequence of +/-1 values.
  2. Let X0 be the vector encoding the target image (the butterfly above). Generate entirely random independent vectors X1, X2, ..., Xn-1 and distribute the corresponding white noise images to the first n-1 students.
  3. Student n is given an image corresponding to the vector Xn obtained by multiplying (coordinate-wise) all the other vectors:
    Xn = X0* X1 * X2 * ... * Xn-1
    where "*" denotes coordinate-wise multiplication.
  4. It is simple arithmetic that X0 = Xn* X1 * X2 * ... * Xn-1. So all students working together possess the information to recover the butterfly image.
  5. On the other hand one can use elementary probability to show that, if one selects any subset of size n-1 of the vectors X1, X2, ..., Xn, then this subset behaves as if it is a statistically independent collection of vectors corresponding to white-noise images. (It suffices to consider just one pixel at a time, and show that the corresponding sequence of n-1 random +/-1 values obey all possible multiplication laws.) So no strict subset of the students has any information at all about the butterfly image.

There are many other ways to implement secret-sharing (Google/Bing/DuckDuckGo the phrase "secret sharing"). But this one is nice for probabilists, because it provides a graphic example of why pairwise independence (independence of any two events taken from a larger collection of events) need not imply complete independence.

July 2021

Mo Tu We Th Fr Sa Su
Jun |  Today  |
         1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31   

Search this blog


Most recent comments

  • Thanks to Martin Emil Jakobsen for pointing out a typo in the example of conditioning on a single ev… by Wilfrid Kendall on this entry
  • The paper includes a nice example of application of a log–normal distribution, which is used to mode… by Wilfrid Kendall on this entry
  • See also their webapp by Wilfrid Kendall on this entry

Blog archive

RSS2.0 Atom
Not signed in
Sign in

Powered by BlogBuilder