statistics

Mathelirium ✓ @mathelirium · 5h Now that we've seen a single random-walk MCMC get trapped in one valley and pretend that's the whole posterior, this scene shows one of the nicest fixes called Parallel Tempering. Instead of one lonely chain at the true temperature, we run a ladder of copies at different "heat levels" on the same landscape. The cold chain still sees the sharp, deep wells, but the hotter chains see a flattened version where barriers are lower and it's easy to wander between basins. Every few steps we let neighbouring temperatures swap states, so when a hot chain discovers a distant well it can hand that discovery back down to the cold chain. Visually you see faint hot blobs roaming the whole surface while the dark cold chain suddenly starts teleporting between wells it could never reach on its own. This is a geometry-aware algorithm. #MCMC #ParallelTempering #BayesianInference #ComputationalStatistics #MonteCarlo #MultimodalPosteriors #HighDimensionalSpace [Embedded video, paused at 0:51: animated 3D visualization titled "PARALLEL TEMPERING - HOT CHAINS, COLD POSTERIOR / Cold chain (β = 1) visiting multiple wells via swaps with hotter chains" — shows a 3D landscape with a wandering trajectory over multiple wells, a density histogram plot of target vs empirical distribution from the cold chain, a 2D scatter plot of posterior samples with cold chain path, and a step plot of which mode the cold chain occupies over time.]

@GSalafatinos: The way I did it was progressively asking it to breakdown the problem into self-contained sub-questions that could be used to make a determination. I'd feed each one to a fresh context gemini and report back. This particular subquestion produced the counter example in response. [Embedded image, mathematical text document:] Question: Behavior of TV Distance for Specific α-Bounded Structures Let Ω be a finite set, |Ω| = d. Let α ∈ (0,1/d]. Let Pi, Qi (i = 1,...,n) be α-bounded distributions on Ω, meaning ∀x ∈ Ω, α ≤ Pi(x) ≤ 1 − α and α ≤ Qi(x) ≤ 1 − α. Let δi = ||Pi − Qi||TV and TVn = ||P⊗n − Q⊗n||TV. We are investigating the conjecture TVn ≤ √(Σδi²) · max{1, log(1/α)}. The binary symmetric case (d = 2, Pi = (1−α, α), Qi = (α, 1−α)) appears not to violate the conjecture. We seek to understand if other structures can lead to a violation, particularly for small α (large d) where the gap between potential χ²-based bounds (~√(n/α)) and the conjecture's log(1/α) factor is largest, but perhaps avoiding the rapid saturation seen in the binary case. Consider the following specific structures (or similar ones designed to probe the interaction of small δi, small α, and tensorization): Structure 1: Uniform Background with Small Perturbation Let α = 1/d. Let Qi = Q = (1/d, 1/d, ..., 1/d) be the uniform distribution (which is α-bounded). Let ε be a small positive value such that α − ε ≥ α is NOT required, but P must still be α-bounded. This requires careful construction. * Example Construction: Let d ≥ 3. Define P by moving mass ε from coordinate 2 to coordinate 1. P = (α+ε, α−ε, α, ..., α). For P to be α-bounded, we need α−ε ≥ α, implying ε ≤ 0. Let's try moving mass from d−1 coordinates to one coordinate. Let P(1) = α + (d−1)ε, P(x) = α − ε for x = 2,...,d. * Check α-bounds: We need α − ε ≥ α ⟹ ε ≤ 0. * This seems difficult. Alternative: Let P be only slightly different from Q. Let P(1) = α+ε', P(2) = α+ε'', ..., ΣP(x) = 1. How small must ε', ε'' be to maintain α ≤ P(x), Q(x) ≤ 1−α? * Consider d = 3, α = 0.1. Q = (0.1, 0.4, 0.5) (Assume non-uniform Q to allow more flexibility). Let P = (0.15, 0.4, 0.45). Here δ = 0.05. α ≤ P(x), Q(x) ≤ 1−α. Structure 2: Non-Uniform Background, Difference at Low Probability Let d ≥ 3. Choose a non-uniform Qi = Q such that Q(1) = α but Q(x) > α for x > 1. Let Pi = P be constructed by modifying Q slightly, primarily changing Q(1) and perhaps one other coordinate Q(j) where Q(j) is large, such that P remains α-bounded and δ = ||P−Q||TV is small. * Example Construction (d = 4, α = 0.1): Q = (0.1, 0.25, 0.3, 0.35). This is α-bounded. Let P = (0.15, 0.25, 0.3, 0.30). P is α-bounded. Here δ = 0.05. The difference is concentrated at x = 1 (where Q is small) and x = 4. Question: For structures like those proposed above (or variations): 1. Can TVn = ||P⊗n − Q⊗n||TV be calculated exactly or tightly bounded (upper and lower) for small values of n (e.g., n = 2,3,4,...) and representative small values of α?

Distributions will come up in Loss Functions in Machine Learning (e.g. XGBoost, LightGBM, CatBoost). Selecting the right Loss Function can often improve performance. Examples: - Poisson is used for count data. - Tweedie for mixed continuous data with many zeros like intermittent demand forecasting problems. [image: histogram showing the Tweedie distribution — a tall spike at zero followed by a right-skewed continuous distribution] "The Tweedie distribution has a point mass at zero before following a 'regular' exponential curve."

Benjamin @bschne · 1h Listened to an interview with Nick Patterson, a mathematician now with the Broad Institute who formerly worked at RenTech, and this bit really stuck with me. The smartest practitioners often use surprisingly simple tools, they're just better at applying the right ones right. [Embedded quote block] It's funny that I think the most important thing to do on data analysis is to do the simple things right. So here's a kind of non-secret about what we did at Renaissance. In my opinion, our most important statistical tool was simple regression with one target and one independent variable. It's the simplest statistical model you can imagine, any reasonably smart high school student can do it. Now we have some of the smartest people around working in our Hedge Fund. We have string theorists we recruited from Harvard. And they're doing simple regression. Is this stupid or pointless? Should we be hiring stupid people and paying them less? And the answer is no. And the reason is nobody tells you what the variables you should be regressing. What's the target? Should you do a non-linear transform before you regress? What's the source? Should you clean your data? Do you notice when your results are obviously rubbish? And so on. And the smarter you are, the less likely you are to make a stupid mistake. And that's why I think you often need smart people who appear to be doing something technically very easy, but actually, usually it's not so easy. We're able to do it carefully and precisely. Nick Patterson on "Talking Machines" "AI Safety and The Legacy of Bletchley Park" (S02E04)