*Fisher Information*

#
All 1 entries tagged

No other Warwick Blogs use the tag *Fisher Information* on entries | View entries tagged *Fisher Information* at Technorati | There are no images tagged *Fisher Information* on this blog

## June 03, 2017

### An example of a random variable with finite Fisher information but infinite entropy

The following notes concern some calculations I have made relating to Zanella *et al* (2017). The considerations laid out below are entirely trivial, but are helpful in making it clear whether or not certain conditions might be logically independent of each other.

Consider a random variable *X* with probability density *f*(*x*) = e^{φ(x)}.

- The
*entropy*of*X*is given by*H*= - ∫ log(*f*(*x*))*f*(*x*) d*x*= - ∫ φ(*x*)*f*(*x*) d*x*= -**E**[ φ(*X*) ]; - One is also interested in something loosely related to
*Fisher information*, namely Ι = ∫ φ'(*x*)^{2}*f*(*x*) d*x*=**E**[ φ'(*X*)^{2}].

#### Question:

Is it possible for Ι to be finite while *H* is infinite?

#### Answer:

**Yes**.

Consider a density *f*(*x*) which is proportional (for large positive *x*) to 1/(*x *log(*x*)^{2}). Consequently φ(*x*) = constant - (log *x *+ 2 log log *x*) for large *x*.

1: Using a change of variable (e^{u} = *x*) it can be shown that *H* = - ∫ log(*f*(*x*)) *f*(*x*) d*x* is infinite. The contribution to the entropy *H* for large *x* is given by - ∫^{∞} log(*f*(*x*)) *f*(*x*) d*x,* controlled by

∫^{∞} (log *x* + 2 log log *x*) d*x */(*x *log(*x*)^{2}) = ∫^{∞} (*u* + 2 log *u*) e* ^{u} *d

*u*/(

*u*

^{2}e

^{u}^{}) ≥ ∫

^{∞}d

*u*/

*u*= ∞.

2: On the other hand elementary bounds show that Ι = ∫ φ'(*x*)^{2} *f*(*x*) d*x* can be finite. The contribution to the "Fisher information" Ι for large *x* is given by - ∫^{∞} φ'(*x*)^{2} *f*(*x*) d*x,* related to

∫^{∞} (1 / *x* + 2 / (*x *log *x*) )^{2} d*x */(*x *log(*x*)^{2}) = ∫^{∞} (1 + 2 / log *x* )^{2} d*x */(*x*^{3} log(*x*)^{2}) < constant × ∫^{∞} d*x */ *x*^{3} < ∞.

An example of such a density (behaving in the required manner at ±∞) is

*f*(*x*) = log(2) |*x*| / ((2+*x*^{2}) (log(2+*x*^{2}))^{2}) .

#### Reference

Zanella, G., Bédard, M., & Kendall, W. S. (2017). A Dirichlet Form approach to MCMC Optimal Scaling. *Stochastic Processes and Their Applications*, to appear, 22pp. http://doi.org/10.1016/j.spa.2017.03.021