All entries for Saturday 03 June 2017

An example of a random variable with finite Fisher information but infinite entropy

The following notes concern some calculations I have made relating to Zanella et al (2017). The considerations laid out below are entirely trivial, but are helpful in making it clear whether or not certain conditions might be logically independent of each other.

Consider a random variable X with probability density f(x) = e^φ(x).

1. The entropy of X is given by H = - ∫ log(f(x)) f(x) dx = - ∫ φ(x) f(x) dx = - E[ φ(X) ];
2. One is also interested in something loosely related to Fisher information, namely Ι = ∫ φ'(x)² f(x) dx = E[ φ'(X)² ].

Question:

Is it possible for Ι to be finite while H is infinite?

Answer:

Yes.

Consider a density f(x) which is proportional (for large positive x) to 1/(x log(x)²). Consequently φ(x) = constant - (log x + 2 log log x) for large x.

1: Using a change of variable (e^u = x) it can be shown that H = - ∫ log(f(x)) f(x) dx is infinite. The contribution to the entropy H for large x is given by - ∫^∞ log(f(x)) f(x) dx, controlled by
∫^∞ (log x + 2 log log x) dx /(x log(x)²) = ∫^∞ (u + 2 log u) e^u du /(u² e^u) ≥ ∫^∞du / u = ∞.

2: On the other hand elementary bounds show that Ι = ∫ φ'(x)² f(x) dx can be finite. The contribution to the "Fisher information" Ι for large x is given by - ∫^∞ φ'(x)² f(x) dx, related to
∫^∞ (1 / x + 2 / (x log x) )² dx /(x log(x)²) = ∫^∞ (1 + 2 / log x )² dx /(x³ log(x)²) < constant × ∫^∞ dx / x³ < ∞.

An example of such a density (behaving in the required manner at ±∞) is

f(x) = log(2) |x| / ((2+x²) (log(2+x²))²) .

Reference

Zanella, G., Bédard, M., & Kendall, W. S. (2017). A Dirichlet Form approach to MCMC Optimal Scaling. Stochastic Processes and Their Applications, to appear, 22pp. http://doi.org/10.1016/j.spa.2017.03.021

Wilfrid Kendall : 03 Jun 2017 16:04 |  Tags: Entropy Fisher Information |  Comments (0) |  Close comments |  Report a problem

A puzzle about inference

Here is a puzzle which I often use as a starter for a course of lectures on probabilistic coupling. The puzzle arose during some research -- I devised it as a simple example to show myself why a particular idea would not work -- and I developed the solution using calculations. In a lunch queue, I asked James Norris of Cambridge whether he could think of a calculation-free answer, and he found one almost immediately.

• You go to Starbucks with a friend.
  
• You are the quiet type, and stay in the front room near the window, sipping a coffee slowly to make it last.
  
• She is extrovert, and has gone through to a further room where the louder people gather.
  There she plays a game: toss n coins, win if r heads or more.
• After the game should have finished, a third friend comes through and says,
  either: "She won the first toss'';
  or: "She won at least one toss''.
  (Why don't people speak more clearly?!)
• Which is better news for your friend? Prove your answer without calculation!

Wilfrid Kendall : 03 Jun 2017 14:24 |  Tags: Coupling Motivation St220 |  Comments (0) |  Close comments |  Report a problem