All entries for Saturday 03 June 2017

June 03, 2017

An example of a random variable with finite Fisher information but infinite entropy

The following notes concern some calculations I have made relating to Zanella et al (2017). The considerations laid out below are entirely trivial, but are helpful in making it clear whether or not certain conditions might be logically independent of each other.

Consider a random variable X with probability density f(x) = eφ(x).

  1. The entropy of X is given by H = - ∫ log(f(x)) f(x) dx = - ∫ φ(x) f(x) dx = - E[ φ(X) ];
  2. One is also interested in something loosely related to Fisher information, namely Ι = ∫ φ'(x)2 f(x) dx = E[ φ'(X)2 ].

Question:

Is it possible for Ι to be finite while H is infinite?

Answer:

Yes.

Consider a density f(x) which is proportional (for large positive x) to 1/(x log(x)2). Consequently φ(x) = constant - (log x + 2 log log x) for large x.

1: Using a change of variable (eu = x) it can be shown that H = - ∫ log(f(x)) f(x) dx is infinite. The contribution to the entropy H for large x is given by - ∫ log(f(x)) f(x) dx, controlled by
(log x + 2 log log x) dx /(x log(x)2) = ∫ (u + 2 log u) eu du /(u2 eu) ≥ ∫du / u = ∞.

2: On the other hand elementary bounds show that Ι = ∫ φ'(x)2 f(x) dx can be finite. The contribution to the "Fisher information" Ι for large x is given by - ∫ φ'(x)2 f(x) dx, related to
(1 / x + 2 / (x log x) )2 dx /(x log(x)2) = ∫ (1 + 2 / log x )2 dx /(x3 log(x)2) < constant × ∫ dx / x3 < ∞.

An example of such a density (behaving in the required manner at ±∞) is

f(x) = log(2) |x| / ((2+x2) (log(2+x2))2) .

Reference

Zanella, G., Bédard, M., & Kendall, W. S. (2017). A Dirichlet Form approach to MCMC Optimal Scaling. Stochastic Processes and Their Applications, to appear, 22pp. http://doi.org/10.1016/j.spa.2017.03.021


A puzzle about inference

Here is a puzzle which I often use as a starter for a course of lectures on probabilistic coupling. The puzzle arose during some research -- I devised it as a simple example to show myself why a particular idea would not work -- and I developed the solution using calculations. In a lunch queue, I asked James Norris of Cambridge whether he could think of a calculation-free answer, and he found one almost immediately.

  • You go to Starbucks with a friend.
  • You are the quiet type, and stay in the front room near the window, sipping a coffee slowly to make it last.
  • She is extrovert, and has gone through to a further room where the louder people gather.
    There she plays a game: toss n coins, win if r heads or more.
  • After the game should have finished, a third friend comes through and says,
    either: "She won the first toss'';
    or: "She won at least one toss''.
    (Why don't people speak more clearly?!)
  • Which is better news for your friend? Prove your answer without calculation!

June 2017

Mo Tu We Th Fr Sa Su
May |  Today  | Jul
         1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30      

Search this blog

Galleries

Most recent comments

  • More recently I have noticed the following elementary fact about convergence in probability. Suppose… by Wilfrid Kendall on this entry
  • Thanks to Martin Emil Jakobsen for pointing out a typo in the example of conditioning on a single ev… by Wilfrid Kendall on this entry
  • The paper includes a nice example of application of a log–normal distribution, which is used to mode… by Wilfrid Kendall on this entry
  • See also their webapp https://045.medsci.ox.ac.uk/ by Wilfrid Kendall on this entry

Blog archive

Loading…
Not signed in
Sign in

Powered by BlogBuilder
© MMXXIV