\section{[Internal use] : Weaknesses people may ask about at LVC}

* Prior on sky is skymap

* Discrete sky grid: biases? What about high SNR? (A: just use adapted resolution; not ready yet)

* Uniform weighting

* Not discarding values after burnin

* Adaptation algorithm: why parameters chosen?

* Computational cost: is this really favorable in CPU-hours?  Are we wasting CPU hours to achieve low-latency that's not needed?

** A: Not clear our overall cost is higher...we may be more accurate than 1000 samples, the usual metric for MCMC...we need a clear comparison

* Rotation of the earth

\subsection{Runtime}

* Might be ram-limited in python below 10-ish Hz (multiple harmonics and IFOs, sampled at 16kHz for 30 min)
%	1byte/sample * 16384*30min *60s/min ~ 28 Mb 

\subsection{Applications}

\textbf{These need to be mentioned on the CBC project page}

* Einstein at home scaling?

* GRB project with Alex (already on table)

* [partial] Port likelihood into lalinference, see if they can get any improvement with it, mainly for precessing systems.

\section{[Internal use] Postprocessing details}
Notes, possibly for integration with main text
\begin{widetext}

\noindent \textbf{Doing the integral: Polar coordinates for linear spoked integration}: Our spoked mass grid is
naturally interpolated in polar coordinates $r,\theta$.  Keeping in mind the change-of-coordinates to align the
principal axes of our error ellipsoid with the data, we can evaluate the evidence integral as 
\begin{align}
(\mc,\eta)_a &= [\sqrt{\Gamma}]_{ab}  (r\cos \theta,r\sin \theta)_b \\
\int d\mc \eta p(\mc,\eta) \LikeRed(\mc,\eta) &= \int |\Gamma| r d\theta \; p(\mc(r,\theta),\eta(r, \theta)) \LikeRed(\ldots)
\end{align}
where the matrix square root of the Fisher matrix is used to change coordinates to (scaled) principal axes of the ellipse:
\[
\sqrt{\Gamma}_{ab} = \sqrt{\gamma}_1 \hat{v}_{1,a}\otimes v_{1,a} + \sqrt{\gamma_2} v_{2b}\otimes v_{2b}
\]
where $\{\gamma_k,v_k\}$ is the eigensystem of $\Gamma$.

\end{widetext}


\section{[Internal use] Monte Carlo integration}

\subsection{Skymaps}
\noindent \textbf{[Skymaps]  Discretizing a volume} We integrate over the sky by translating a continuous distribution into
discrete, equal-area pixels.  Specifically, we start with the abstract process of breaking an integral over a volume
$\Omega$ in the sky coordinates ($\gamma$)  up into nonoveralapping
volumes $A_n$ with $\cup A_n = \Omega$  , where the extrinsic volumes are small enough that we can treat any expected
function (e.g., $L,p$) can be approximated as nearly independent of $\gamma$ in $A_n$:
\begin{widetext}
\begin{align}
\LikeRed(\lambda)
& = \int_{V\times\Omega}\Like(\lambda,\theta,\gamma)p(\theta)p(\gamma) d \theta d\gamma 
%\\& 
=  \sum_{n=1}^N \int_{A_n} [\int_V  L(\gamma,\theta,\gamma) p(\theta) ] p(\gamma)d\gamma 
%\\ & 
 \simeq  \sum_{n=1}^N  \left [\int_V  L(\gamma,\theta,\gamma_n) p(\theta) \right] P_n 
\end{align}\end{widetext}
where $P_n \equiv \int_{A_n} d\gamma p(\gamma)$ is the prior probability for the volume $A_n$, and $\gamma_n$ is a
representative point in $A_n$.   Note
\begin{eqnarray}
\sum_n P_n =1
\end{eqnarray}
We are \emph{particularlly} interested in uniform distributions (equal-area tilings of the sky), so 
\[
P_n = \frac{1}{N} \qquad   \text{Area}(A_n) = \frac{4\pi}{N}
\]

\noindent \emph{Monte Carlo integration with discrete volumes}: As above, but apply to Monte Carlo integration, with a
sampling prior $p_s(\theta)p_s(\gamma)$ in the sky $\gamma$ and all other extrinsic variables $\theta$:
\begin{align}
\LikeRed(\lambda)
& \simeq  \sum_{n=1}^N \int_{A_n} \left [\int_V  L(\gamma,\theta,\gamma_n) p(\theta)p(\gamma) \right]
\end{align}
Proceed as above, but approximate $p(\gamma_n)/p_s(\gamma_n) \simeq P_n/P_{s,n}$, to find
\begin{align}
\LikeRed(\lambda)
& =\sum_n \left[\int_{V}\Like(\lambda,\theta,\gamma_n)\frac{p(\theta)}{p_s(\theta)}p_s(\theta)d\theta \right] \frac{ P_n }{P_{s,n}} P_{s,n}
\end{align}
Now imagine performing a Monte Carlo integral in this form.   Monte Carlo integration corresponds to selecting a random
integer $n$ and a random sample $\theta$ drawn from $p_s(\theta)$.  Indexing the $N$ choices as $n_{\alpha}$ and
$\theta_\alpha$, we find
\begin{align}
\LikeRed(\lambda)
& = \sum_{\alpha}\frac{1}{N}
\Like(\lambda,\theta_{\alpha},\gamma_{n,\alpha})\frac{p(\theta_\alpha)}{p_s(\theta_\alpha)} \frac{ P_{n_\alpha}
}{P_{s,n}} 
\end{align}
In other words, when performing a Monte Carlo integral over a discrete subset (e.g., skymap), we use the
\emph{integrated} probabilities $P_n$ as our ``PDF''.

\emph{Concrete discrete random sampling}: For a discrete uniform equal-area tiling, we will use a
\textbf{discrete} probability distribution, not a continuous one.   Specifically, our code 
\begin{itemize}
\item \emph{Prior PDF}: Since we always assume uniform on the sky,  provide $P_n=1/N$ at the $n$th
point (i.e., given the sky position of the $n$th tile), \emph{not} $p(\gamma_n)$.

\item \emph{Skymap sampling PDF}: Bayestar provides normalized weighted $s_{s,n}$ already.  
%% After fixing a
%%   \textbf{normalization issue} (below), we use
%% \[
%% P_{s,n} = s_n \frac{N'}{N}
%% \]
%% where $N,N'$ are defined below.

\item \emph{Skymap random samples via CDF inverse}: We generate random integers $n$ using random numbers on $[0,1]$, finding the
  closest integer $n$ to the  cumulative of $P_n$; see the text.
\end{itemize}




\noindent \textbf{[Skymaps] Finite-area skymaps and truncation}: We will use skymaps with
$p_s(\gamma)=0$  in a volume $U$ where $p(\gamma)\ne 0$.  With restricted support, we can't integrate
integrate generic functions $L$ with $L>0$ with $p_s=0$.  To be concrete, if we split up the integral over the sky ($\Omega$) into $U$ and
$\Omega-U$, we are ignoring the contribution from $\Omega-U$:
\begin{align}
\LikeRed(\lambda) &= \int_U d\gamma p_s(\gamma) 
 [\int d\theta p_s(\theta)
  \frac{p(\theta)p(\gamma)}{p_s(\theta)p_s(\gamma)}\Like] 
  \nonumber \\ &
 + \int_{\Omega-U}d\gamma p(\gamma) p(\theta)\Like 
\nonumber \\ &
\simeq \int_U d\gamma p_s(\gamma) [\int d\theta p_s(\theta)
  \frac{p(\theta)p(\gamma)}{p_s(\theta)p_s(\gamma)}\Like] 
\end{align}
The second step introduces systematic error, assumed small.

The second integral is a conventional Monte Carlo integral over a region $U$ using normalized sampling distributions $p_s$ and
can be approximated in the usual way.

Note the ratio of $p/p_s$ \emph{automatically} includes any ``normalization factor'' associated with only drawing
samples from $U$ rather than from the whole volume $V$

%% , rescaling the total number of samples $N_s$ drawn from $p_s$ by the
%% ratio of volumes of the prior and of $p_s$ over that region $U$
%% \begin{eqnarray}
%% N_{\rm corr} = N / \int_U p_s(\gamma) d\gamma
%% \end{eqnarray}
%% We will apply this correction to the returned values of $p_s$, returning ``rescaled'' probabilities 
%% % mcsampler.py  : self._renorm  in pseudo_pdf  and self._expand_valid_points.
%% \begin{eqnarray}
%% p_s \rightarrow \frac{p_s}{\int_U p(\gamma)  d\gamma}
%% \end{eqnarray}
%% Specifically,  if $s_n$ are the  skymap weights reported by \BS, normalized so $\sum_n s_n=1$, we use an ``effective''
%% sampling ``probability'' $P_{s,n}$ of
%% \begin{eqnarray}
%% P_{s,n} = s_n \frac{N'}{N}
%% \end{eqnarray}
%% where $N'$ is the number of nonzero samples in $s$ and $N$ is the number of pixels in $s$.

\noindent \textbf{[Skymaps] Subtle implementation issues: cutoffs and lookup tables}: As described in the text, we use a
discretized lookup table.  As not described in the text, this table de facto \emph{discards} points with weights the
quantized probability.   For real skymaps, the choice of probability quantum $P_{\rm min}$ can be what determines
precisely what points are sampled and hence what part of $\int L p d\theta d\gamma$ are are not sampling \emph{at all}.

As a concrete example, with $n=12*64^2$ sky points, a uniform weighting gives each pixel $P_{n}
=1/n \simeq 2\times 10^{-5}$.  A qunatum of probability $P_{\rm min} $ should \emph{definitely} be smaller than this, to
allow the option of recovering a uniform skymap.
%
More broadly, a skymap with a peak and a wide tail can have that tail truncated if $P_{\rm min}$ is too small.  For
example, if $5\%$ of the skymap probability is distributed across a wide area $\Omega$ of the skymap, we need $P_{\rm
  min} < 0.05 (4\pi/\Omega)/n$, to be sure that area is sampled.  
%
For safety, ROS recommends adopting a small probability quantum $\frac{10}{N}<P_{\rm min} < 0.01/n$, where $n$ is the
number of sky pixels and $N$ is the number of iterations in ILE.\footnote{This choice ensures that when ILE is run, at
  least a few draws occur even from the lowest probability pixels.  Does this matter?  Principally for tests, or if
we expect the \BS{} skymap might be wrong.}  \textbf{This choice requires self-consistent use of
  multiple code parameters, some currently hardcoded. }

Once chosen, the quantum of probability defines what pixes de facto have zero probability and are never sampled. 
%% \noindent \emph{Convergence issues and sampling bounds on  small probability quanta}: Small probability quanta introduce their own
%% potential problems.  The smaller the probability quantum, the more likely you are to have a pixel with $P_{s,n} \simeq
%% P_{\rm min}$.  On average, that pixel will be sampled every  $1/P_{\rm min}$ sky iterations -- a number that might be in
%% excess of all samples drawn in one ILE run!
Ideally, the likelihood will be zero in  low-probability pixels, no matter what $\lambda,\theta$ are, circumventing
any issues with undersampling and convergence rate.


\subsection{Monte Carlo integration}
\textbf{Non-uniform, variance-weighted weighting}: By default we combine results with different sampling priors treating
them equally (e.g., as if they have converged to the same sampling prior) when evaluating the integral at each mass
point.  

That doesn't minimize variance and can be particularly
ineffective when one ILE run has a problem.  Really, we should use variance-reducing weighting: if $x=ax_1 +b x_2$ with $a+b=1$ and
$\left<x_1\right> =\left<x_2\right> =\left<x\right>$, we can minimize the variance of $x$ by choosing weight $a$ to
minimize $a^2 \sigma_1^2 + (1-a)^2 \sigma_2^2$, so 
\begin{eqnarray}
x = \frac{\sigma_2^2}{\sigma_1^2+\sigma_2^2} x_1 +  \frac{\sigma_1^2}{\sigma_1^2+\sigma_2^2} x_2
\end{eqnarray}
A straightforward generalization exists with arbitrary numbers of weights:
\[
x = \frac{x_k/\sigma_k^2}{\sum_q 1/\sigma_q^2}
\]
We could and should implement this scheme when recombining results at each mass point. 
%% a^2 s1^2 + b^2 s2^2 + (1 - a - b)^2 s3^3
%% Solve[{D[%, a] == 0, D[% == 0, b]}, {a, b}]

\begin{widetext}
\noindent \textbf{Error estimates for $P(<x)$: Single-job}:  ILE reports on 1d cumulative distributions $P(<x)$ from
each job.  Both the numerator and denominator are Monte Carlo integrals using $N$ samples, allowing us to generate an
error estimate.  Keeping in mind the samples are independent and identically distributed, we can evaluate the standard
deviation of the numerator and denominator in the large-$N$ limit
\begin{align}
\hat{P} &= \frac{\sum_k w_k \theta(x-x_k)}{\sum_k w_k} \equiv \frac{ I(<x)}{I} \\
\left< \hat{P} \right> &\simeq \frac{1}{I} \sum_k \left<w_k \theta(x-x_k) \right>  \\
\left<I(<x)^2\right>& \equiv \left<[\sum_k w_k \theta(x-x_k)]^2\right> 
  = \sum_k[ \left<w_k^2 \theta(x-x_k)\right>  -\left<w_k\theta(x-x_k)\right>^2
  + \left<\sum_{k} w_k \theta(x-x_k)\right>^2]  \\
\text{estimate: }\sigma_{I(x)}^2 &= \sum_k w_k^2 \theta(x-x_k)  - I^2 P(x)^2 \\
\text{estimate: } \sigma_{I} &= \sum_k \left<w_k^2\right> - I^2 \\
\sigma_P^2 & \simeq \frac{\sum_k w_k^2 \theta(x-x_k)}{I^2} - P(x)^2 + (\text{value at } x_{\rm max})
\end{align}
\end{widetext}
where the last expression adds errors in quadrature from the numerator and denominator, assuming \emph{uncorrelated}
errors.    (In fact, the two are correlated \editremark{fixme: important for scaling the error as $P(1-P)$ })



\subsection{Marginalizing over parameter}

\noindent \textbf{Skymap marginalization: A proposal}: A discrete skymap usually has relatively concentrated support -- roughly
$10^3$ pixels or less, say.  Rather than perform Monte Carlo integration to approximate $\int P_k
\Like(\lambda,\theta_k)$ over the sky, we can \emph{directly} integrate, via a discrete operation:


\emph{Is this a good idea?}: This procedure requires evaluating the likelihood on  \emph{every pixel in the skymap}, so
is of considerable advantage only for very
narrowly targeted (3-IFO, high-SNR) sky localizations.  In such a narrowly-focused case, an adaptive integrator will almost certainly
discover the best-fitting single sky position.  That said, if we're worried about \emph{strongly-separated peaks}, this
procedure will provide an (expensive) backup.

\section{[Internal use] Results: Targeted studies}

\begin{itemize}
\item Single event, 100 noise realizations, plus comparison with Fisher matrix

\item Zero-spin 2015 MDC: [\textbf{Not started}]  For completeness, we propose to reproduce the 2015 MDC,  using
  injections with exactly zero spin.

\end{itemize}

\subsection{Detailed investigation of one event}
Sample run results: Single event, with full DAG, done multiple times (for complete consistency/reproducibility)

Sample run results: extrinsic parameters (for one and several noise realizations)

Sample run results: $L_{\rm red}(\mc,eta)$ (for one and several noise realizations). Expected accuracy at each mass
point.. Translating into  $p(m_1,m_2)$ using a uniform mass prior.

Comparison with Fisher and MCMC


\begin{figure}
\caption{\label{fig:TargetedEvent:LikelihoodVersusMchirpEta}\textbf{Targeted study: Posterior distribution
    versus component masses, for several noise realizations}: : The 90\% confidence interval derived from $L_{\rm red}$.  For
comparison, the prediction from a Fisher matrix is shown as a solid black curve.
 \textbf{PLACEHOLDER/INTENT}
}
\end{figure}


\section{[Internal use] Results: Additional production-environment investigations}

\begin{itemize}
\item 2016 BNS MDC: [\textbf{Not started}]

\end{itemize}

\section{[Internal use] Incomplete or not-yet-implemented investigations}

\subsection{Measures of convergence}

Measures to assess whether we're done
\begin{itemize}
\item * Integral: Classic MC error estimate (central limit); reproducibility across runs and via subsamples (e.g., chisquared)

\item * 1d posterior:  Monte carlo error (see elsewhere); $n_{\rm eff}$; $L^2$ or KL divergence between subsamples or
  across runs, compared to statistics.

\item * sampling distribution: similarly
\end{itemize}


\subsection{Semianalytic marginalization over distance}


\section{[Internal use] Future directions}

Not for distribution!

\subsection{Short GRBs}

Alex

\subsection{Adding dimensions}

Tides [Les]

Aligned spin

\subsection{Even more speed}

Because the precompute phase is \emph{much} faster than marginalizing over extrinsic parameters, we could accelerate
code performance in a number of ways, potentially enabling us to tackle higher dimensions

\noindent \textbf{Einstein at home}: Transmit the precomputed data (10ms of timeseries and a few scalars) to the
Einstein at home grid, giving each distant host (say) 20 mass points to handle.  With $10^5$ cores, that means roughly
$20\times10^5/10$ intrinsic points per day...enough to handle precessing spin faster than any other extant method.

\noindent \textbf{Interpolate $Q,U$}: Perform the precompute step several times, building up a grid for $Q$ and $U,V$,

\subsection{Tigher search integration}


\noindent \textbf{Search filters as input}
The searches provide filter outputs.  We ought to be able to use those as inputs to construct $Q_{k,lm}$ for any
$\lambda,l,m$, via a suitable lookup table.   This could completely eliminate the startup/precompute cost.



\noindent \textbf{As detection statistic?}: To what extent is this approach useful compared to coherent PTF?  We can
also do timeslides at a fixed sky location.


\section{[Internal use] Results: Targeted studies}

\begin{itemize}
\item Single event, 100 noise realizations, plus comparison with Fisher matrix

\item Zero-spin 2015 MDC: [\textbf{Not started}]  For completeness, we propose to reproduce the 2015 MDC,  using
  injections with exactly zero spin.

\end{itemize}

\section{[Internal use] Results: Targeted studies}

\begin{itemize}
\item Single event, 100 noise realizations, plus comparison with Fisher matrix

\item Zero-spin 2015 MDC: [\textbf{Not started}]  For completeness, we propose to reproduce the 2015 MDC,  using
  injections with exactly zero spin.

\end{itemize}

\subsection{Detailed investigation of one event}
Sample run results: Single event, with full DAG, done multiple times (for complete consistency/reproducibility)

Sample run results: extrinsic parameters (for one and several noise realizations)

Sample run results: $L_{\rm red}(\mc,eta)$ (for one and several noise realizations). Expected accuracy at each mass
point.. Translating into  $p(m_1,m_2)$ using a uniform mass prior.

Comparison with Fisher and MCMC



\section{[Internal use] Results: Additional production-environment investigations}

\begin{itemize}
\item 2016 BNS MDC: [\textbf{Not started}]

\end{itemize}

\section{[Internal use] Incomplete or not-yet-implemented investigations}

\subsection{Measures of convergence}

Measures to assess whether we're done
\begin{itemize}
\item * Integral: Classic MC error estimate (central limit); reproducibility across runs and via subsamples (e.g., chisquared)

\item * 1d posterior:  $n_{\rm eff}$; $L^2$ or KL divergence between subsamples or across runs

\item * sampling distribution: similarly
\end{itemize}


\subsection{Semianalytic marginalization over distance}


\section{[Internal use] Future directions}

Not for distribution!

\subsection{Short GRBs}


\section{[Internal use] Incomplete or not-yet-implemented investigations}

\subsection{Measures of convergence}

Measures to assess whether we're done
\begin{itemize}
\item * Integral: Classic MC error estimate (central limit); reproducibility across runs and via subsamples (e.g., chisquared)
\end{itemize}

\section{[Internal use] Future directions}

Not for distribution!

\subsection{Short GRBs}

Alex

\subsection{Adding dimensions}

Tides [Les]

Aligned spin

\subsection{Even more speed}

Because the precompute phase is \emph{much} faster than marginalizing over extrinsic parameters, we could accelerate

\section{[Internal use] Patches pending or desired}

Persons/proposing supporting  a change listed as [name].

\noindent \textbf{Next push}

* Bugfix on ILE MC variance

* Uniform on sky prior with bayestar skymaps

\noindent \textbf{Other important updates}

* [ROS] Add monte carlo error to $P(<x)$ estimate plots (optional) and description in text

*  [ROS]  $b_k \ne 0$ for any sky position (else complications arise re normalization).

* [ROS] Interpolation of $L$ vs masses using spokes (higher order) -- current plots are mathematica

\noindent \textbf{Minor code tweaks}

* [ROS] Larger nchunk

* [ROS] Change tempering exponent definition from $(Lp/p_s)^\beta$ to $L^\beta p/p_s$, to facilitate analysis

* [ROS] Variance weighting when recombining samples from different runs at the same mass point


\noindent \textbf{Infrastructure}

* [ROS] Change DAG (number of jobs, max iterations): current solution with skymap is overkill


* [ROS] For automated end-to-end tests, add \gstlal{} coinc generation and \BS{} skymaps to \texttt{stage\_injections}

** Approximate solution: use bayestar tool to generate ``fake'' coincs and skymaps from sims:
\href{https://www.lsc-group.phys.uwm.edu/ligovirgo/cbcnote/ParameterEstimationModelSelection/BAYESTARHowTo}{Bayestar docs}

* [ROS] Add posterior in mchirp, eta accounting for prior.

* [Patrick] Logging

* [Richard/Chris] Report hashtag in log and process\_params

* [ROS] Warnings about implicit cutoffs re skymaps (\texttt{min\_p} plus the \texttt{cdf} target, both of which define
regions to be ignored).



* [ROS] Modularized version that lets us change mass quickly (e.g., to loop over mass points; to test mass points
against known-good extrinsic parameters; etc)
%load all wights stored 
%loop and eval likelihood with stored samples only
%useful for exploring mass space using fixed extrinsic grid set by previous 
%only need a few samples to get an idea


\noindent \textbf{Exploration}

* [ROS] Region library (i.e. ellipsoids): test if in; perform integral (mcsampler); return points (grid/unstructured)

** mcsampler: return dump of params, so we can construct marginalized posteriors/PDFs using usual infrastructure

* [ROS] Refinement stage: a single job after a mass point, using the best-suited

\section{Methods  paper 2: Spin}

Adding the spin dimensions is important and worth a detailed study.


Very early; keep thoughts organized in outline, below


\begin{itemize}

  \item Introduction
  \item Revised methods
    \begin{itemize}
        \item  $h_{lm}$ in $J$ frame
       \item  Single spin: approximate harmonic method (treat each sideband $e^{im\alpha+im\gamma}h_{22}^{ROT}$ as a
         seperate filter, and $\beta$ as an extrinsic parameter.  De facto means adding only one additional dimension
         ($\kappa$) over aligned spin;
         all else extrinsic)
        \item Spin dynamics and dimensional reduction/reorganization.  ``PN resonance coordinates.''
        \item Role of prior for spin (6d vs 3d vs 2d prior for 2spin, 1spin, aligned spin) has effect
        \item New approximately extrinsic parameters (e.g., $\alpha$ as approximately extrinsic)
    \end{itemize}

 \item Discussion and sample results: Aligned spin
     
   * single event in detail: 3d posterior; evidence for spin vs zero spin

 \item Discussion and results: Single precessing spin

   * ``fixed $\beta$

   * single event in detail

 \item Discussion and results: Double spin

  * challenge of higher dimensions
\end{itemize}
