#LyX 2.3 created this file. For more info see http://www.lyx.org/
\lyxformat 544
\begin_document
\begin_header
\save_transient_properties true
\origin unavailable
\textclass scrreprt
\begin_preamble
\usepackage[dvipsnames]{xcolor}
\usepackage{fourier}

%\definecolor{MyGreen}{HTML}{CBD90F}
%\definecolor{MyYellow}{HTML}{DFEF11}
%\definecolor{MyOrange}{HTML}{FEC34D}

\definecolor{MyYellow}{HTML}{D8D9E9}
%\definecolor{MyGreen}{HTML}{B3B4DA}
\definecolor{MyOrange}{HTML}{FFC57A}
\definecolor{MyGreen}{HTML}{EBEBF1}



\newcommand{\nwayversion}{1.0}
\newcommand{\nway}{\textsc{Nway}}
\newcommand{\examplebox}[1]{\colorbox{MyGreen}{\begin{minipage}{10.3cm}#1\end{minipage}}}
\newcommand{\otherbox}[1]{\colorbox{MyYellow}{\begin{minipage}#1\end{minipage}}}
\newcommand{\code}[1]{{\texttt{#1}}}
\newcommand{\apj}{ApJ}
\newcommand{\apjs}{ApJS}
\newcommand{\aap}{A\&A}
\newcommand{\mnras}{MNRAS}
%\newcommand{\mycaution}[1]{{\large \danger } \#1}
\newcommand{\mycaution}[1]{{\Huge \danger } \colorbox{MyOrange}{\begin{minipage}{8cm}#1\end{minipage}}}

%\newcommand{\mycaution}[1]{\colorbox{Green}{\begin{minipage}#1\end{minipage}}}
%\newcommand{\caution}[1]{\colorbox{Red}{\begin{minipage}{\huge \danger} #1\end{minipage}}}



\IfFileExists{lucimatx.sty}{
  % ----- if Lucida availalable------------------
  \usepackage[
    scale=0.875,
    stdmathitalics=true,
    stdmathdigits=true]{lucimatx}
 \linespread{1.02}
}{
  % ------ if Lucida not available --------------
  \usepackage{times}
  \usepackage{amssymb}
}
\end_preamble
\options 10pt,openany
\use_default_options true
\maintain_unincluded_children false
\language english
\language_package default
\inputencoding auto
\fontencoding global
\font_roman "default" "default"
\font_sans "default" "default"
\font_typewriter "default" "default"
\font_math "auto" "auto"
\font_default_family default
\use_non_tex_fonts false
\font_sc false
\font_osf false
\font_sf_scale 100 100
\font_tt_scale 100 100
\use_microtype false
\use_dash_ligatures true
\graphics default
\default_output_format default
\output_sync 0
\bibtex_command default
\index_command default
\paperfontsize default
\spacing single
\use_hyperref true
\pdf_bookmarks true
\pdf_bookmarksnumbered false
\pdf_bookmarksopen false
\pdf_bookmarksopenlevel 1
\pdf_breaklinks false
\pdf_pdfborder false
\pdf_colorlinks true
\pdf_backref false
\pdf_pdfusetitle true
\pdf_quoted_options "linkcolor=blue,citecolor=black"
\papersize a5paper
\use_geometry true
\use_package amsmath 1
\use_package amssymb 1
\use_package cancel 1
\use_package esint 1
\use_package mathdots 1
\use_package mathtools 1
\use_package mhchem 1
\use_package stackrel 1
\use_package stmaryrd 1
\use_package undertilde 1
\cite_engine natbib
\cite_engine_type authoryear
\biblio_style plain
\use_bibtopic false
\use_indices false
\paperorientation portrait
\suppress_date false
\justification true
\use_refstyle 1
\use_minted 0
\index Index
\shortcut idx
\color #008000
\end_index
\topmargin 1.5cm
\bottommargin 1.7cm
\secnumdepth 3
\tocdepth 3
\paragraph_separation indent
\paragraph_indentation default
\is_math_indent 0
\math_numbering_side default
\quotes_style english
\dynamic_quotes 0
\papercolumns 1
\papersides 1
\paperpagestyle default
\tracking_changes false
\output_changes false
\html_math_output 0
\html_css_as_file 0
\html_be_strict false
\end_header

\begin_body

\begin_layout Title
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 - The versatile catalogue matching tool
\end_layout

\begin_layout Subtitle
User Manual, version 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nwayversion
\end_layout

\end_inset


\end_layout

\begin_layout Author
Written by Johannes Buchner
\end_layout

\begin_layout Date
22.
 August 2016
\end_layout

\begin_layout Standard
\begin_inset CommandInset toc
LatexCommand tableofcontents

\end_inset


\end_layout

\begin_layout Chapter
Why 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway
\end_layout

\end_inset

?
\end_layout

\begin_layout Standard
In astrophysics, a common task is to assemble multi-wavelength information
 about individual sources.
 This is done by taking the detections of sources in the sky (positions,
 errors, and fluxes/magnitudes) from a catalogue of one wavelength and matching
 it to another from another wavelength, or multiple such catalogues.
 Care has to be taken to consider all possible matches and also the possibility
 that the source does not have a counterpart in a catalogue of a given depth.
 For many classes of sources, the Spectral Energy Distribution (SED) provides
 additional hints, which associations are likely real.
 For instance, the color distribution of stars in the WISE bands is different
 than that of quasars or galaxies.
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 is a generic solution to these tasks:
\end_layout

\begin_layout Enumerate
Matching of N catalogues simultaneously.
\end_layout

\begin_layout Enumerate
Consideration of all combinatorically possible matches.
\end_layout

\begin_layout Enumerate
Consideration of partial matches across catalogues, i.e.
 the absence of counterparts in some catalogues.
\end_layout

\begin_layout Enumerate
Taking into account the positional uncertainty.
\end_layout

\begin_layout Enumerate
Computation of a probability for each possible match.
\end_layout

\begin_layout Enumerate
Computation of a probability that there is no match.
\end_layout

\begin_layout Enumerate
Incorporating magnitude, color or other information about the sources of
 interest, refining the match probabilities.
\end_layout

\begin_layout Section*
Contributors
\end_layout

\begin_layout Itemize
Mara Salvato – idea and leading the science case.
\end_layout

\begin_layout Itemize
Tamás Budavári – shared basic implementation of his formulae.
\end_layout

\begin_layout Itemize
Sotiria Fotopoulou – initial implementation for matching three catalogues.
\end_layout

\begin_layout Itemize
Johannes Buchner – complete code rewrite for the general case, documentation,
 manual and adding features.
\end_layout

\begin_layout Standard
The code is the results of many discussion among colleagues and friends.
 We thank in particular: Tamás Budavári, Sotiria Fotopoulou, Fabrizia Guglielmet
ti, Arne Rau, Tom Dwelly, Andrea Merloni and Kirpal Nandra.
\end_layout

\begin_layout Section*
How to read this manual
\end_layout

\begin_layout Standard
This manual walks you through the installation and usage.
 It is best to try out the examples as you read, because they illustrate
 how 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 works.
\end_layout

\begin_layout Standard
Yellow boxes indicate text about the illustrative examples and their explanation
s.
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
colorbox{MyYellow}{
\end_layout

\end_inset

 
\begin_inset Box Frameless
position "t"
hor_pos "c"
has_inner_box 1
inner_pos "t"
use_parbox 0
use_makebox 0
width "100col%"
special "none"
height "1in"
height_special "totalheight"
thickness "0.4pt"
separation "3pt"
shadowsize "4pt"
framecolor "black"
backgroundcolor "none"
status open

\begin_layout Plain Layout
Useful explanation for users on how to prepare files or how to run 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 are found in blue boxes.
\end_layout

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
examplebox{
\end_layout

\end_inset

Light blue boxes indicate examples you can run yourself to learn 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway
\end_layout

\end_inset

.
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
Sometimes there are 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{commands}
\end_layout

\end_inset

 you can copy-paste.
\end_layout

\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
If you are interested in the rigorous mathematical details, go to Chapter
\begin_inset space ~
\end_inset


\begin_inset CommandInset ref
LatexCommand ref
reference "chap:math"

\end_inset

.
\end_layout

\begin_layout Section*
Terminology
\end_layout

\begin_layout Paragraph
Source 
\end_layout

\begin_layout Standard
A detection in a certain wavelength significant enough to be recorded in
 a catalogue with sky position.
\end_layout

\begin_layout Paragraph
Counterpart
\end_layout

\begin_layout Standard
The corresponding source in another catalogue.
\end_layout

\begin_layout Paragraph
Association
\end_layout

\begin_layout Standard
A specific combination of entries from the various catalogues, i.e.
 a tuple of detections associated with each other.
\end_layout

\begin_layout Paragraph
Match
\end_layout

\begin_layout Standard
Same as association, used interchangeably.
\end_layout

\begin_layout Paragraph
Object
\end_layout

\begin_layout Standard
A physical entity in the real universe, emitting radiation.
\end_layout

\begin_layout Chapter
User Manual
\end_layout

\begin_layout Section
Installation
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 is a pure Python program.
 Install 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 through
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\backslash
$ sudo pip install nway}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
This will give you the tool 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{nway.py}
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
Or if you do not have root access:
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\backslash
$ pip install nway -{}-user}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
The tool 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{nway.py}
\end_layout

\end_inset

 is installed for you as 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\backslash
textasciitilde/.local/bin/nway.py}
\end_layout

\end_inset

.
 
\end_layout

\begin_layout Standard
If that directory is in your 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\backslash
$PATH}
\end_layout

\end_inset

, you can run 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\backslash
$ nway.py -{}-help}
\end_layout

\end_inset

.
\end_layout

\begin_layout Paragraph
Development version
\end_layout

\begin_layout Standard
To get the latest development version, fetch 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 from 
\end_layout

\begin_layout Standard
\begin_inset Box Boxed
position "t"
hor_pos "c"
has_inner_box 1
inner_pos "t"
use_parbox 0
use_makebox 0
width "100col%"
special "none"
height "1in"
height_special "totalheight"
thickness "0.4pt"
separation "3pt"
shadowsize "4pt"
framecolor "black"
backgroundcolor "none"
status open

\begin_layout Plain Layout
\begin_inset Flex URL
status open

\begin_layout Plain Layout

https://github.com/JohannesBuchner/nway
\end_layout

\end_inset


\end_layout

\end_inset


\end_layout

\begin_layout Standard
and run in the directory
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\backslash
$ python setup.py install -{}-user}
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
You need to install these python packages:
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{scipy, astropy, matplotlib, progressbar-latest, argparse, joblib, healpy}
\end_layout

\end_inset

.
\end_layout

\begin_layout Paragraph
Upgrading from older 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 versions
\end_layout

\begin_layout Standard
To upgrade, uninstall the older 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 version first: 
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\backslash
$ pip uninstall nway}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Repeat until it says that 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 is not installed.
 Then follow the installation instructions above.
\end_layout

\begin_layout Section
Citing 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 correctly
\end_layout

\begin_layout Standard
Please cite Salvato et al., MNRAS, 2018 (ArXiV: 
\begin_inset Flex URL
status open

\begin_layout Plain Layout

https://arxiv.org/abs/1705.10711
\end_layout

\end_inset

, ADS: 
\begin_inset Flex URL
status open

\begin_layout Plain Layout

http://adsabs.harvard.edu/abs/2018MNRAS.473.4937S
\end_layout

\end_inset

).
\end_layout

\begin_layout Section
Development, questions and issues
\end_layout

\begin_layout Standard
Please let us know if you have comments about this manual or problems running/in
stalling 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway
\end_layout

\end_inset

.
 
\end_layout

\begin_layout Standard
For reporting bugs or requesting features, 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway
\end_layout

\end_inset

's issue tracker is at the following location.
 
\end_layout

\begin_layout Standard
\begin_inset Box Boxed
position "t"
hor_pos "c"
has_inner_box 1
inner_pos "t"
use_parbox 0
use_makebox 0
width "100col%"
special "none"
height "1in"
height_special "totalheight"
thickness "0.4pt"
separation "3pt"
shadowsize "4pt"
framecolor "black"
backgroundcolor "none"
status open

\begin_layout Plain Layout
\begin_inset Flex URL
status open

\begin_layout Plain Layout

https://github.com/JohannesBuchner/nway
\end_layout

\end_inset


\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 is a small but powerful tool.
 Most questions or apparent issues arise to understand which information
 lead to a counterpart being preferred, and therefore understanding the
 input data well.
 This manual will guide you through the computation and the pieces of informatio
n added together.
\end_layout

\begin_layout Section
Best practice matching
\begin_inset CommandInset label
LatexCommand label
name "sec:Best-practice-matching"

\end_inset


\end_layout

\begin_layout Standard
To achieve reliable results, we recommend that you run matching with increasing
 amount of information (distance-based first, §
\begin_inset CommandInset ref
LatexCommand ref
reference "sec:Simple-distance-based-matching"

\end_inset

, then adding priors on source properties §
\begin_inset CommandInset ref
LatexCommand ref
reference "chap:additional-priors"

\end_inset

) and understand how each influences the results.
\end_layout

\begin_layout Standard
Typically, the goal is a compilation of 
\begin_inset Quotes eld
\end_inset

best matches
\begin_inset Quotes erd
\end_inset

, i.e.
 choosing one reliable counterpart for each source.
 Whatever method used, there are always false selection fractions and false
 non-selection fractions in play, which should be characterized.
 To this end, we recommend to shift the source catalogues by a distance
 much larger than the positional errors to simulate the results for chance
 alignment.
 This fake catalogue should not coincide with original positions (tool 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

nway-create-fake-catalogue.py
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 may help).
 For the matching run with this fake catalogue, use 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 with the same settings and from the output, choose cut-off limits (
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_any
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

) that correspond to the desired false selection fraction (tool 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

nway-calibrate-cutoff.py
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 may help).
 The output of your first 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 run advises how to use these tools to characterise false selection rates
 and to find an appropriate 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_any
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 threshold.
\end_layout

\begin_layout Standard
Then the 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 output on the real data can be truncated based on this criterion (
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_any>cutoff
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

), and made into a best match catalogue (
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

match_flag==1
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

).
 It may be worth noting ambiguous cases with multiple solutions and to store
 these secondary solutions with similar probabilities in another catalogue
 (
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

match_flag==2
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

).
\end_layout

\begin_layout Standard
\begin_inset Newpage newpage
\end_inset


\end_layout

\begin_layout Section
Simple distance-based matching
\begin_inset CommandInset label
LatexCommand label
name "sec:Simple-distance-based-matching"

\end_inset


\end_layout

\begin_layout Standard
Before exploring the full power of 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway
\end_layout

\end_inset

, we consider a simple, illustrative case.
 We have three catalogues, provided as FITS files:
\end_layout

\begin_layout Standard
\begin_inset Float figure
placement H
wide false
sideways false
status open

\begin_layout Plain Layout

\emph on
Input:
\end_layout

\begin_layout Plain Layout
\begin_inset Tabular
<lyxtabular version="3" rows="5" columns="5">
<features tabularvalignment="middle">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Primary Catalogue
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2nd Catalogue
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3rd Catalogue
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
A
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\alpha$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\mathbb{A}$
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
B
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\beta$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\mathbb{B}$
\end_inset


\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" bottomline="true" leftline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\begin_layout Plain Layout

\emph on
\begin_inset VSpace 0.1cm
\end_inset


\end_layout

\begin_layout Plain Layout

\emph on
Output:
\end_layout

\begin_layout Plain Layout
\begin_inset Tabular
<lyxtabular version="3" rows="11" columns="5">
<features tabularvalignment="middle">
<column alignment="center" valignment="top" width="2cm">
<column alignment="center" valignment="top" width="2cm">
<column alignment="center" valignment="top" width="2cm">
<column alignment="center" valignment="top">
<column alignment="center" valignment="top">
<row>
<cell alignment="center" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Primary Catalogue Entry
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
2nd Catalogue Entry
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
3rd Catalogue Entry
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" topline="true" bottomline="true" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
Probability
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
A
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\alpha$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\mathbb{A}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell multirow="3" alignment="center" valignment="middle" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
A group
\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
A
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\alpha$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\mathbb{B}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell multirow="4" alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
A
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\alpha$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(none)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell multirow="4" alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
A
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\beta$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\mathbb{A}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell multirow="4" alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
A
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\beta$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\mathbb{B}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell multirow="4" alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
A
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\beta$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(none)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell multirow="4" alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
A
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(none)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\mathbb{A}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell multirow="4" alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
A
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(none)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
\begin_inset Formula $\mathbb{B}$
\end_inset


\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell multirow="4" alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
A
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(none)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
(none)
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell multirow="4" alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout

\end_layout

\end_inset
</cell>
</row>
<row>
<cell alignment="center" valignment="top" leftline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
B
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" rightline="true" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
...
\end_layout

\end_inset
</cell>
<cell alignment="center" valignment="top" usebox="none">
\begin_inset Text

\begin_layout Plain Layout
B group
\end_layout

\end_inset
</cell>
</row>
</lyxtabular>

\end_inset


\end_layout

\end_inset


\end_layout

\begin_layout Standard
In 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway
\end_layout

\end_inset

, only the first catalogue (
\series bold
the primary catalogue
\series default
) plays a special role.
 For each entry of it, counterparts are sought from the other catalogues.
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
colorbox{MyYellow}{
\end_layout

\end_inset


\begin_inset Box Frameless
position "t"
hor_pos "c"
has_inner_box 1
inner_pos "t"
use_parbox 0
use_makebox 0
width "100col%"
special "none"
height "1in"
height_special "totalheight"
thickness "0.4pt"
separation "3pt"
shadowsize "4pt"
framecolor "black"
backgroundcolor "none"
status open

\begin_layout Subsection
Example - Preparing input files
\end_layout

\begin_layout Plain Layout
Note these points about preparing a catalogue input file:
\end_layout

\begin_layout Enumerate
Each catalogue needs to be a FITS file.
 The second extension should be the table (first extension is a header).
 TOPCAT writes files in this way.
\begin_inset Newline newline
\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
examplebox{
\end_layout

\end_inset

Three example catalogues are provided for you in the 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{doc/}
\end_layout

\end_inset

 directory: 
\series bold
COSMOS_IRAC.fits, COSMOS_OPTICAL.fits and COSMOS_XMM.fits
\series default
.
 These are the same files as in Appendix B of 
\begin_inset CommandInset citation
LatexCommand cite
key "2018MNRAS.473.4937S"
literal "true"

\end_inset

, extracted from 
\begin_inset CommandInset citation
LatexCommand cite
key "Sanders2007"
literal "true"

\end_inset

/
\begin_inset CommandInset citation
LatexCommand cite
key "McCracken2007"
literal "true"

\end_inset

, 
\begin_inset CommandInset citation
LatexCommand cite
key "Ilbert2010"
literal "true"

\end_inset

 and 
\begin_inset CommandInset citation
LatexCommand cite
key "Brusa2010"
literal "true"

\end_inset

 respectively.
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Enumerate
The data table needs to have a extension name and the keyword SKYAREA.
 The extension name is used as a prefix as all columns are copied to the
 output catalogue.
 The SKYAREA keyword tells the area on the sky in 
\series bold
square degrees
\series default
 covered by the catalogue.
 This is important for estimating the chance of random alignments.
 You can use the tool 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python nway-write-header.py mycat.fits mytablename myskyarea}
\end_layout

\end_inset

 to set the fits header.
 
\begin_inset Newline newline
\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
examplebox{
\end_layout

\end_inset

For our example files we have a optical, IRAC and XMM catalogue covering
 2 square degrees: 
\begin_inset Newline newline
\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python nway-write-header.py COSMOS
\backslash
_OPTICAL.fits OPT 2}
\end_layout

\end_inset


\begin_inset Newline newline
\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python nway-write-header.py COSMOS
\backslash
_IRAC.fits IRAC 2}
\end_layout

\end_inset


\begin_inset Newline newline
\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python nway-write-header.py COSMOS
\backslash
_XMM.fits XMM 2}
\end_layout

\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Enumerate
Each catalogue needs to have a column RA and DEC providing the coordinates
 in 
\series bold
degrees
\series default
.
 To make your life easier, 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 tries to be a bit fuzzy and detect the columns named RA_something etc.
 It will print out which columns it found and used.
\end_layout

\begin_layout Enumerate
The primary catalogue needs to have a ID column.
 In our example this is the X-ray catalogue.
 To make your life easier, 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway
\end_layout

\end_inset

 tries to be a bit fuzzy and detect the columns named ID_something etc.
 It will print out which columns it found and used.
\end_layout

\begin_layout Enumerate
Otherwise the file can have arbitrary columns which are copied over to the
 output file.
\end_layout

\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
Every possible combination of association is considered.
 However, in practice you do not want an extremely large output catalogue
 with extremely distant, unlikely to be physically associated.
 You can set the largest distance in degrees to consider by setting 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-radius}
\end_layout

\end_inset

.
 This speeds up the computation.
 But use a value that is much larger than the largest positional error.
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
colorbox{MyYellow}{
\end_layout

\end_inset


\begin_inset Box Frameless
position "t"
hor_pos "c"
has_inner_box 1
inner_pos "t"
use_parbox 0
use_makebox 0
width "100col%"
special "none"
height "1in"
height_special "totalheight"
thickness "0.4pt"
separation "3pt"
shadowsize "4pt"
framecolor "black"
backgroundcolor "none"
status open

\begin_layout Subsection
Example - Matching two catalogues
\end_layout

\begin_layout Plain Layout
Lets try the simplest example and match the XMM X-ray catalogue to an optical
 catalogue.
 The XMM catalogue has a pos_err column with the positional error in arcseconds.
 For the optical catalogue we will assume a fixed error of 0.1 arcseconds.
\end_layout

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
examplebox{
\end_layout

\end_inset

Run this command in the doc/ folder:
\end_layout

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway.py COSMOS
\backslash
_XMM.fits :pos
\backslash
_err COSMOS
\backslash
_OPTICAL.fits 0.1 -{}-out=example1.fits -{}-radius 15 -{}-prior-completeness
 0.9}
\end_layout

\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
Lets understand what we put in:
\end_layout

\begin_layout Enumerate
We passed two catalogue files: COSMOS_XMM.fits and COSMOS_OPTICAL.fits.
 For the first one, we told 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway
\end_layout

\end_inset

 to use the column (
\begin_inset Quotes eld
\end_inset

:
\begin_inset Quotes erd
\end_inset

) 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

pos_err
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 in that catalogue for the positional error (
\series bold
always in arcsec
\series default
).
 For the second one we specified a fixed error of 0.1 arcsec.
\end_layout

\begin_layout Enumerate
We specified where the output should be written (
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-out}
\end_layout

\end_inset

).
\end_layout

\begin_layout Enumerate
The largest XMM error is 7.3 arcsec, so we adopt a cropping radius of 15
 arcsec to speed up the matching (
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-radius 15}
\end_layout

\end_inset

).
 A larger radius produces a more complete catalogue.
 For dense catalogues larger radii can be much slower to compute, as the
 number of combinations to consider rises exponentially.
\end_layout

\begin_layout Enumerate
The parameter 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-prior-completeness 0.9}
\end_layout

\end_inset

 is mentioned below.
\end_layout

\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset Float figure
placement H
wide false
sideways false
status open

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
colorbox{MyYellow}{
\end_layout

\end_inset


\begin_inset Box Frameless
position "t"
hor_pos "c"
has_inner_box 1
inner_pos "t"
use_parbox 0
use_makebox 0
width "100col%"
special "none"
height "1in"
height_special "totalheight"
thickness "0.4pt"
separation "3pt"
shadowsize "4pt"
framecolor "black"
backgroundcolor "none"
status open

\begin_layout Plain Layout
Lets understand what 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 did:
\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Enumerate
\begin_inset listings
lstparams "basicstyle={\scriptsize\ttfamily},tabsize=2"
inline false
status open

\begin_layout Plain Layout

NWAY arguments:
\end_layout

\begin_layout Plain Layout

    catalogues:  COSMOS_XMM.fits, COSMOS_OPTICAL.fits
\end_layout

\begin_layout Plain Layout

    position errors/columns:  :pos_err, 0.1
\end_layout

\begin_layout Plain Layout

      from catalogue "XMM" (1797), density is 3.706579e+07
\end_layout

\begin_layout Plain Layout

      from catalogue "OPT" (560536), density is 1.156188e+10
\end_layout

\begin_layout Plain Layout

    magnitude columns:  
\end_layout

\end_inset

It reads the catalogues and looks at their densities.
\end_layout

\begin_layout Enumerate
\begin_inset listings
lstparams "basicstyle={\scriptsize\ttfamily},tabsize=2"
inline false
status open

\begin_layout Plain Layout

matching with 15.000000 arcsec radius
\end_layout

\begin_layout Plain Layout

matching: 1007283192 naive possibilities
\end_layout

\begin_layout Plain Layout

matching: hashing
\end_layout

\begin_layout Plain Layout

    using RA  columns: RA, RA
\end_layout

\begin_layout Plain Layout

    using DEC columns: DEC, DEC
\end_layout

\begin_layout Plain Layout

matching: healpix hashing on pixel resolution ~ 18.036304 arcsec (nside=8192)
\end_layout

\begin_layout Plain Layout

100% | 562333|############################################|Time: 0:00:13
\end_layout

\begin_layout Plain Layout

matching: collecting from 61787 buckets, creating cartesian products ...
\end_layout

\begin_layout Plain Layout

100%|61787|###############################################|Time: 0:00:02
\end_layout

\begin_layout Plain Layout

matching: 462267 unique matches from cartesian product.
 sorting ...
\end_layout

\begin_layout Plain Layout

merging in 10 columns from input catalogues ...
\end_layout

\begin_layout Plain Layout

100% 10|##################################################|Time: 0:00:00
\end_layout

\begin_layout Plain Layout

    adding angular separation columns
\end_layout

\begin_layout Plain Layout

matching:  22435 matches after filtering by search radius
\end_layout

\end_inset

Within 20 seconds it created a cross-match of remotely possible associations
 (1,007,283,192 in principle, 22,435 within 15 arcseconds).
\end_layout

\begin_layout Enumerate
It found ID, RA, DEC, and positional error columns.
\end_layout

\begin_layout Enumerate
\begin_inset listings
lstparams "basicstyle={\scriptsize\ttfamily},tabsize=2"
inline false
status open

\begin_layout Plain Layout

Computing distance-based probabilities ...
\end_layout

\begin_layout Plain Layout

  finding position error columns ...
\end_layout

\begin_layout Plain Layout

    Position error for "XMM": found column XMM_pos_err: Values are [0.109000..7.3010
00]
\end_layout

\begin_layout Plain Layout

    Position error for "OPT": using fixed value 0.100000
\end_layout

\begin_layout Plain Layout

  finding position columns ...
\end_layout

\begin_layout Plain Layout

  building primary_id index ...
\end_layout

\begin_layout Plain Layout

  computing probabilities ...
\end_layout

\begin_layout Plain Layout

      correcting for unrelated associations ...
 not necessary
\end_layout

\begin_layout Plain Layout

\end_layout

\begin_layout Plain Layout

Computing final probabilities ...
\end_layout

\begin_layout Plain Layout

    grouping by column "XMM_ID" and flagging ...
\end_layout

\begin_layout Plain Layout

100%|  1797|###################################|Time: 0:00:00
\end_layout

\end_inset

It computed the probability of each association.
\end_layout

\begin_layout Enumerate
\begin_inset listings
lstparams "basicstyle={\scriptsize\ttfamily},tabsize=2"
inline false
status open

\begin_layout Plain Layout

creating output FITS file ...
\end_layout

\begin_layout Plain Layout

    writing "example1.fits" (37836 rows, 17 columns)
\end_layout

\end_inset

It wrote the output file 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

example1.fits
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

.
 This file contains all columns from the input catalogues and the computed
 probabilities (see below for their meaning).
\end_layout

\end_inset


\end_layout

\begin_layout Standard
So how does 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 deal with a particular, possible association and compute its probability?
 
\end_layout

\begin_layout Standard
The probability of a given association is computed by comparing the probability
 of a random chance alignment of unrelated sources (prior) to the likelihood
 that the source is the same.
 The gory mathematical details are laid out in Section 
\begin_inset CommandInset ref
LatexCommand ref
reference "sec:math"

\end_inset

, but from a user point of view the following is important:
\end_layout

\begin_layout Enumerate
The chance of a random alignment depends on the source sky density of the
 various catalogues.
 
\series bold
So each catalogue needs to have a FITS header entry 
\family typewriter
SKYAREA
\family default
 which tells the area covered by the catalogue in square degrees.

\series default
 The source density on the sky is then computed by the number of entries
 divided by that area.
 You can use the tool 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python nway-write-header.py mycat.fits mytablename myskyarea}
\end_layout

\end_inset

 to set the fits header.
\end_layout

\begin_layout Enumerate
Varying depths between the catalogues and different coverage can further
 reduce the fraction of expected matches.
 This can be adjusted by setting 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-prior-completeness=0.9}
\end_layout

\end_inset

, if previous experience is that only 90% of sources have a match with the
 given inputs.
\end_layout

\begin_layout Standard
The outputs catalogue then contains six important new columns along with
 all columns of the input catalogues:
\end_layout

\begin_layout Enumerate
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

dist_bayesfactor
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

: logarithm of ratio between prior and posterior from distance matching
\end_layout

\begin_layout Enumerate
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

dist_post
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

: Distance posterior probability comparing this association vs.
 no association, as in 
\begin_inset CommandInset citation
LatexCommand citet
key "Budavari2008"
literal "true"

\end_inset

.
\end_layout

\begin_layout Enumerate
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_single
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

: Same as 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

dist_post
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 unless additional information was added, see Section
\begin_inset space ~
\end_inset


\begin_inset CommandInset ref
LatexCommand ref
reference "chap:additional-priors"

\end_inset

.
 
\end_layout

\begin_layout Enumerate

\series bold
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_any
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\series default
: For each entry in the primary catalogue (e.g.
 A) the probability that one of the association is the correct one is computed.
 Because every catalogue is limited by its depth, it is possible that the
 true counterpart has not been found yet.
 Our testing suggest that the 
\series bold
threshold for a secure catalogue depends on the application.
 
\series default
Section
\begin_inset space ~
\end_inset


\begin_inset CommandInset ref
LatexCommand ref
reference "sec:Best-practice-matching"

\end_inset

 explains how to calibrate a threshold.
\end_layout

\begin_layout Enumerate

\series bold
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_i
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\series default
: For each possible association for each entry in the primary catalogue
 (e.g.
 A), the relative probability is computed.
 Our testing suggest that secure, pure catalogue
\series bold
 should keep only associations where 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_i>=0.1
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

.
 Secondary solutions down to 0.1 may be interesting.
 These thresholds may depend on the application – please report what your
 testing gives.
\series default

\begin_inset Newline newline
\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
mycaution{
\end_layout

\end_inset

Low 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_any
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 and 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_i
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 values by themselves do not necessarily mean that the counterpart is ruled
 out.
 It can also mean that there is not enough evidence/information to declare
 it a counterpart.
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Enumerate

\series bold
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

match_flag
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\series default
: The most probable match is indicated with 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

1
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 for each primary catalogue entry.
 Secondary, almost as good solutions are marked with 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

2
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

.
 By default, the maximum allowed ratio is at most 0.5, but the user can modify
 this threshold via the 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-acceptable-prob}
\end_layout

\end_inset

 parameter.
 All other associations are marked with 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

0
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
Use the last three columns to identify sources with one solution, possible
 secondary solutions, and to build final catalogues.
 Chapter 
\begin_inset CommandInset ref
LatexCommand ref
reference "sec:math"

\end_inset

 explains how these quantities are computed.
 To filter out low-probability associations (low 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_i
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

) from the output catalogue, the 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-min-prob}
\end_layout

\end_inset

 parameter can be used.
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
colorbox{MyYellow}{
\end_layout

\end_inset


\begin_inset Box Frameless
position "t"
hor_pos "c"
has_inner_box 1
inner_pos "t"
use_parbox 0
use_makebox 0
width "100col%"
special "none"
height "1in"
height_special "totalheight"
thickness "0.4pt"
separation "3pt"
shadowsize "4pt"
framecolor "black"
backgroundcolor "none"
status open

\begin_layout Subsection
Example - Output of matching two catalogues
\end_layout

\begin_layout Plain Layout
Lets understand the output fits file and the associations found for a particular
 X-ray source.
 
\end_layout

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
examplebox{
\end_layout

\end_inset

Open the fits file and find XMM_ID=60388.
 As you can see from the 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_i
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 column, this is a ambiguous case, where more than one optical counterpart
 is possible.
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
Below is an illustration of this ambiguous case (produced with 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway-explain.py example1.fits 60388}
\end_layout

\end_inset

).
\end_layout

\begin_layout Plain Layout
Two sources are at a similar distance from the X-ray source (blue, with
 error circle).
 Therefore their association probability (
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_i
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

) is similar.
 The slightly higher one is marked as match_flag=1 (orange), the other with
 2 (yellow).
\end_layout

\begin_layout Plain Layout
Section 
\begin_inset CommandInset ref
LatexCommand ref
reference "chap:additional-priors"

\end_inset

 solves this by adding more information (the magnitude distribution).
 But we can also solve this another way.
 We know AGN (the X-ray source) emit in the infrared, so you can also match
 with an IRAC catalogue.
 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
examplebox{
\end_layout

\end_inset

Make a three-way match like so:
\end_layout

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway.py COSMOS
\backslash
_XMM.fits :pos
\backslash
_err COSMOS
\backslash
_OPTICAL.fits 0.1 COSMOS
\backslash
_IRAC.fits 0.5 -{}-out=example3.fits -{}-radius 15}
\end_layout

\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
However, overall we should note that 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_any
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 is low, indicating that probably neither of the two candidates is the counterpa
rt.
\end_layout

\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset Float figure
placement h
wide false
sideways false
status open

\begin_layout Plain Layout
\begin_inset Graphics
	filename example1.fits_explain_60388.pdf
	width 100col%

\end_inset


\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset Newpage clearpage
\end_inset


\end_layout

\begin_layout Section
Matching with additional information
\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "chap:additional-priors"

\end_inset


\end_layout

\begin_layout Standard
For many classes of sources, the Spectral Energy Distribution (SED) provides
 additional hints, which associations are likely real.
 For instance, bright X-ray sources have a different color distribution
 in the WISE bands than non-X-ray emitting objects.
 A powerful feature of 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 is to take advantage of this additional information to improve the matching.
 Section 
\begin_inset CommandInset ref
LatexCommand ref
reference "sec:mag-priors"

\end_inset

 has the mathematical details and a comparison to the Likelihood Ratio method.
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
colorbox{MyYellow}{
\end_layout

\end_inset


\begin_inset Box Frameless
position "t"
hor_pos "c"
has_inner_box 1
inner_pos "t"
use_parbox 0
use_makebox 0
width "100col%"
special "none"
height "1in"
height_special "totalheight"
thickness "0.4pt"
separation "3pt"
shadowsize "4pt"
framecolor "black"
backgroundcolor "none"
status open

\begin_layout Subsection
Example - Using magnitude information
\end_layout

\begin_layout Plain Layout
X-ray sources (which we are looking for in our example) have a different
 optical magnitude distribution than non-X-ray emitting objects.
 Lets take advantage of this information:
\end_layout

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
examplebox{
\end_layout

\end_inset

Run this command: 
\end_layout

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway.py COSMOS
\backslash
_XMM.fits :pos
\backslash
_err COSMOS
\backslash
_OPTICAL.fits 0.1 -{}-out=example2.fits -{}-radius 15 -{}-prior-completeness
 0.9 -{}-mag OPT:MAG auto -{}-mag-radius 3.5}
\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
The last two parts are new: 
\end_layout

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-mag OPT:MAG auto -{}-mag-radius 3.5}
\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
We use the column 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

MAG
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 from the catalogue 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

OPT
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 (FITS table name), therefore 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-mag OPT:MAG}
\end_layout

\end_inset

.
 After this follows that the magnitude prior histogram should be generated
 from the data (mode 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

auto
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

), by comparing the 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{MAG}
\end_layout

\end_inset

 histogram of sources within 3.5 arcsec of a X-ray source (
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-mag-radius}
\end_layout

\end_inset

) to that of full histogram.
\end_layout

\begin_layout Plain Layout

\emph on
(example continued below)
\end_layout

\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Standard
There are three possible ways to specify the prior in 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway
\end_layout

\end_inset

: In all cases you specify 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-mag column-name [filename|auto]}
\end_layout

\end_inset

.
 You can use 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-mag}
\end_layout

\end_inset

 several times.
\end_layout

\begin_layout Enumerate
\begin_inset Quotes eld
\end_inset

File-mode
\begin_inset Quotes erd
\end_inset

: If we know the magnitude distribution of X-ray detected AGN we can provide
 this prior distribution as a table (histogram).
 This table contains the color histogram of the sources of interest (X-ray
 detected AGN) and a histogram of other, field sources (more details below
 
\begin_inset CommandInset ref
LatexCommand vpageref
reference "par:Providing-prior-file"

\end_inset

).
\end_layout

\begin_layout Enumerate
\begin_inset Quotes eld
\end_inset

Simple auto-mode
\begin_inset Quotes erd
\end_inset

: Specifying 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{auto}
\end_layout

\end_inset

 instead of a file name derives the two distributions from the data, as
 we did in our example: All sources inside 3.5 arcseconds (
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-mag-radius}
\end_layout

\end_inset

 parameter) of a X-ray source are put into one histogram, and all others
 into another histogram.
\end_layout

\begin_layout Enumerate
\begin_inset Quotes eld
\end_inset

Bayesian auto-mode
\begin_inset Quotes erd
\end_inset

: Bayesian distance probabilities (
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

dist_post
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

) will be used if you leave out 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-mag-radius}
\end_layout

\end_inset

.
 This is in general safer and recommended.
 In small catalogues the histogram may not be sufficiently filled, in which
 case 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 will give a warning (more details below 
\begin_inset CommandInset ref
LatexCommand vpageref
reference "par:auto-mode"

\end_inset

).
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
colorbox{MyYellow}{
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset Box Frameless
position "t"
hor_pos "c"
has_inner_box 1
inner_pos "t"
use_parbox 0
use_makebox 0
width "100col%"
special "none"
height "1in"
height_special "totalheight"
thickness "0.4pt"
separation "3pt"
shadowsize "4pt"
framecolor "black"
backgroundcolor "none"
status open

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
examplebox{
\end_layout

\end_inset

Lets look at the histograms it computed.
 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 created 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{OPT
\backslash
_MAG
\backslash
_fit.pdf}
\end_layout

\end_inset

, and also 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{OPT
\backslash
_MAG
\backslash
_fit.txt}
\end_layout

\end_inset

 as a histogram file:
\end_layout

\begin_layout Plain Layout
\begin_inset Graphics
	filename OPT_MAG_fit.pdf
	width 80col%
	BoundingBox 0bp 198bp 507bp 398bp
	clip

\end_inset


\end_layout

\begin_layout Plain Layout
They are clearly different: Lower magnitude (bright) sources are more likely
 associated to X-ray sources.
 This will help our matching.
\end_layout

\begin_layout Plain Layout
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Plain Layout
As an example, we show below the ambiguous case from before.
 The upper association has been selected because it has a better match by
 magnitude, resolving the ambiguity.
\end_layout

\begin_layout Plain Layout
\begin_inset Graphics
	filename example2.fits_explain_60388.pdf
	width 80col%

\end_inset


\end_layout

\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Paragraph
Multiple priors
\end_layout

\begin_layout Standard
You can specify as many priors as you like, taking advantage of more and
 more information.
 Just repeat 
\begin_inset ERT
status open

\begin_layout Plain Layout

-{}-mag
\end_layout

\end_inset

.
\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
examplebox{
\end_layout

\end_inset

The following example uses one prior from the optical catalogue and another
 prior from the IRAC catalogue.
 A three-way match is performed.
 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway.py COSMOS
\backslash
_XMM.fits :pos
\backslash
_err COSMOS
\backslash
_OPTICAL.fits 0.1 COSMOS
\backslash
_IRAC.fits 0.5  -{}-out=example3.fits -{}-radius 20 -{}-prior-completeness
 0.9 -{}-mag OPT:MAG auto -{}-mag IRAC:mag
\backslash
_ch1 auto  -{}-mag-radius 3.5}
\end_layout

\end_inset


\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Paragraph
Providing a prior as a file
\begin_inset CommandInset label
LatexCommand label
name "par:Providing-prior-file"

\end_inset


\end_layout

\begin_layout Standard
In the paper we demonstrate the use of a WISE magnitude of X-ray sources.
 If such prior information comes from previous studies, the distributions
 can be passed to 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 as a ASCII table histogram.
 This table contains the histogram of the sources of interest (X-ray sources)
 and a histogram of other sources (non-X-ray sources).
 The file 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{OPT
\backslash
_MAG
\backslash
_fit.txt}
\end_layout

\end_inset

 is an example of such a input file, and can be used via 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-mag OPT:MAG OPT
\backslash
_MAG
\backslash
_fit.txt}
\end_layout

\end_inset

.
 It contains four columns (lower and upper bin edge, density of selected
 and non-selected) and looks like this (# indicates comments):
\end_layout

\begin_layout Standard
\begin_inset listings
lstparams "basicstyle={\footnotesize\ttfamily}"
inline false
status open

\begin_layout Plain Layout

# OPT_MAG_fit.txt
\end_layout

\begin_layout Plain Layout

# lo         hi          selected   others
\end_layout

\begin_layout Plain Layout

  10.76000   18.98571    0.00870    0.00183
\end_layout

\begin_layout Plain Layout

  18.98571   20.27286    0.05562    0.01448
\end_layout

\begin_layout Plain Layout

  ...
\end_layout

\end_inset


\end_layout

\begin_layout Standard
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
mycaution{
\end_layout

\end_inset

Keep in mind that a prior created from a different data set can only be
 used if it is applicable to the present data set.
 For example, in the introduction of the paper 
\begin_inset CommandInset citation
LatexCommand cite
key "2018MNRAS.473.4937S"
literal "true"

\end_inset

 we stress that a prior from a comparable X-ray exposure depth must be used
 when deriving color distributions.
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset


\end_layout

\begin_layout Paragraph
A general approach
\end_layout

\begin_layout Standard
Providing priors is not limited to magnitude distributions, you can use
 colors or any other information you want (e.g.
 morphology, variability, etc.).
 The approach is very general, 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 just looks at the corresponding bin and reweighs the probabilities.
 For example, in 
\begin_inset CommandInset citation
LatexCommand cite
key "2018MNRAS.473.4937S"
literal "true"

\end_inset

, the counterparts to ROSAT sources where found using WISE.
 The prior was build by using the color-magnitude (W1-W2 vs W2) properties
 of ~3000 secure counterparts to the 3XMM-Bright survey cut at the depth
 reached by ROSAT.
 
\end_layout

\begin_layout Paragraph
Discovering a prior from distance matching
\begin_inset CommandInset label
LatexCommand label
name "par:auto-mode"

\end_inset


\end_layout

\begin_layout Standard
If you set 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-mag OPT:MAG auto}
\end_layout

\end_inset

 and do not set 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-mag-radius}
\end_layout

\end_inset

, 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 uses the Bayesian distance matching for discovering the histogram of 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{OPT:MAG}
\end_layout

\end_inset

, as follows:
\end_layout

\begin_layout Enumerate
Those with 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

dist_post>0.9
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 are considered safe matches and are used for the 
\begin_inset Quotes eld
\end_inset

selected
\begin_inset Quotes erd
\end_inset

 histogram.
\end_layout

\begin_layout Enumerate
Those with 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

dist_post<0.01
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 are considered safe non-matches and are used for the 
\begin_inset Quotes eld
\end_inset

others
\begin_inset Quotes erd
\end_inset

 histogram.
\end_layout

\begin_layout Enumerate
Entries of -99 are always ignored.
 It is usually better to assign -99 where the magnitude error is large,
 to get cleaner histograms.
\end_layout

\begin_layout Standard
This is in general more cautious, and recommended for large catalogues 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
mycaution{
\end_layout

\end_inset

However, if you only have a small catalogue you may build a poorly sampled
 histogram, potentially leading to biases.
 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 will warn you when only few sources were selected.
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

.
\end_layout

\begin_layout Chapter
Program Arguments
\end_layout

\begin_layout Enumerate
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway.py -{}-help}
\end_layout

\end_inset

 ...
 display help page
\end_layout

\begin_layout Enumerate
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway.py catalogue1 catalogues ...}
\end_layout

\end_inset

 ...
 input catalogues as FITS files.
\end_layout

\begin_layout Enumerate
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway.py -{}-out}
\end_layout

\end_inset

 ...
 Output file name (also a FITS file).
\end_layout

\begin_layout Enumerate
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway.py -{}-radius}
\end_layout

\end_inset

 This radius (in degrees) is used to discard distant pairs.
 Always choose a value that is much larger than the largest positional uncertain
ty, then this value will not change the results.
 Smaller values make the code run faster and use less memory by reducing
 the number of combinations to explore.
\end_layout

\begin_layout Enumerate
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway.py -{}-prior-completeness 1}
\end_layout

\end_inset

 ...
 set expected matching completeness (default: 1)
\end_layout

\begin_layout Enumerate
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway.py -{}-mag MAGCOLUMN MAGFILE}
\end_layout

\end_inset

 ...
 name of <table>:<column> for magnitude biasing, and file name for magnitude
 histogram (use auto for auto-computation within mag-radius).
 Example: 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{-{}-mag OPT:MAG auto -{}-mag IRAC:mag
\backslash
_irac1 irac
\backslash
_histogram.txt}
\end_layout

\end_inset


\end_layout

\begin_layout Enumerate
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway.py -{}-mag-radius}
\end_layout

\end_inset

 ...
 If set, and a auto prior is defined, then the selected sources are taken
 from within this radius of the primary sources (in arc seconds).
 If not set (recommended), the Bayesian posterior from distance matching
 is used, which incorporates positional errors.
\end_layout

\begin_layout Enumerate
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway.py -{}-min-prob}
\end_layout

\end_inset

 ...
 only retain associations in the output catalogue exceeding this 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_i
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 value.
 Recommended: 0.1.
\end_layout

\begin_layout Enumerate
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{python ../nway.py -{}-acceptable-prob}
\end_layout

\end_inset

 ...
 affects the flagging of secondary solutions (
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

match_flag
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 column).
 If the secondary is within this difference (default: 0.005), it is marked
 as a secondary solution.
\end_layout

\begin_layout Chapter
Input file specifications and units
\end_layout

\begin_layout Enumerate
Each catalogue needs to be a FITS file.
 The second extension should be the table (first extension is a header).
 
\end_layout

\begin_layout Enumerate
The data table needs to have a extension name.
\end_layout

\begin_layout Enumerate
The header of the data table needs the keyword SKYAREA, which specifies
 the area covered by the catalogue in 
\series bold
square degrees
\series default
.
\end_layout

\begin_layout Enumerate
Each catalogue needs to have a column RA and DEC 
\series bold
in degrees
\series default
.
 To make your life easier, 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 tries to be a bit fuzzy and detect the columns named RA_something etc.
 
\end_layout

\begin_layout Enumerate
The primary catalogue needs to have a ID column.
 To make your life easier, 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway~
\end_layout

\end_inset

 tries to be a bit fuzzy and detect the columns named ID_something etc.
 
\end_layout

\begin_layout Enumerate
Positional error columns, if used, need to be 
\series bold
in arcseconds
\series default
.
 
\end_layout

\begin_layout Standard
Example catalogues are provided in the 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{doc/}
\end_layout

\end_inset

 directory: COSMOS_IRAC.fits, COSMOS_OPTICAL.fits and COSMOS_XMM.fits.
\end_layout

\begin_layout Chapter
Mathematical details and Implementation
\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "chap:math"

\end_inset


\end_layout

\begin_layout Section
Distance-based matching
\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "sec:math"

\end_inset


\end_layout

\begin_layout Standard
Lets consider the problem of finding counterparts to a primary catalogue
 (
\begin_inset Formula $i=1$
\end_inset

), in our example for the X-ray source position catalogue.
 Let each 
\begin_inset Formula $N_{i}$
\end_inset

 denote the number of entries for the catalogues used, and 
\begin_inset Formula $\nu_{i}=N_{i}/\Omega_{i}$
\end_inset

 denote their source surface density on the sky.
 
\end_layout

\begin_layout Standard
If a counterpart is required to exist in each of the 
\begin_inset Formula $k$
\end_inset

 catalogues, there are 
\begin_inset Formula $\prod_{i=1}^{k}N_{i}$
\end_inset

 possible associations.
 If we assume that a counterpart might be missing in each of the matching
 catalogues, there are 
\begin_inset Formula $N_{1}\cdot\prod_{i=2}^{k}(N_{i}+1)$
\end_inset

 possible associations.
 This minor modification, negligible for 
\begin_inset Formula $N_{i}\gg1$
\end_inset

, is ignored in the following for simplicity, but handled in the code.
\end_layout

\begin_layout Standard
If each catalogue covers the same area with some respective, homogeneous
 source density 
\begin_inset Formula $\nu_{i}$
\end_inset

, the probability of a chance alignment on the sky of physically unrelated
 objects can then be written 
\begin_inset CommandInset citation
LatexCommand citep
after "eq. 25"
key "Budavari2008"
literal "true"

\end_inset

 as 
\begin_inset Formula 
\begin{equation}
P(H)=N_{1}/\prod_{i=1}^{k}N_{i}=1/\prod_{i=2}^{k}N_{i}=1/\prod_{i=2}^{k}\nu_{i}\Omega_{i}.
\end{equation}

\end_inset

Thus 
\begin_inset Formula $P(H)$
\end_inset

 is the prior probability of an association.
 The posterior should strongly exceed this prior probability, to avoid false
 positives.
 
\end_layout

\begin_layout Standard
To account for non-uniform coverage, 
\begin_inset Formula $P(H)$
\end_inset

 is modified by a 
\begin_inset Quotes eld
\end_inset

prior completeness factor
\begin_inset Quotes erd
\end_inset

 
\begin_inset Formula $c$
\end_inset

, which gives the expected fraction of sources with reliable counterpart
 (due to only partial coverage of the matching catalogues 
\begin_inset Formula $\Omega_{i>1}\neq\Omega_{1}$
\end_inset

, depth of the catalogues and/or systematic errors in the coordinates).
 Our prior can thus be written as 
\begin_inset Formula 
\begin{equation}
P(H)=c/\prod_{i=2}^{k}\nu_{i}\Omega_{1}.\label{eq:prior}
\end{equation}

\end_inset


\end_layout

\begin_layout Standard
Bayes' theorem connects the prior probability 
\begin_inset Formula $P(H)$
\end_inset

 to the posterior probability 
\begin_inset Formula $P(H|D)$
\end_inset

, by incorporating information gained from the observation data 
\begin_inset Formula $D$
\end_inset

 via 
\begin_inset Formula 
\begin{equation}
P(H|D)\propto P(H)\times P(D|H).\label{eq:bayes}
\end{equation}

\end_inset


\end_layout

\begin_layout Standard
We now extend the approach of 
\begin_inset CommandInset citation
LatexCommand citet
key "Budavari2008"
literal "true"

\end_inset

, to allow matches where some catalogues do not participate in a match.
 Comparing A12 and A14 in 
\begin_inset CommandInset citation
LatexCommand citet
key "Budavari2008"
literal "true"

\end_inset

, assuming that positions lie on the celestial sphere and adopting the expansion
s developed in their Appendix B, we can write down likelihoods.
 For a counterpart across 
\begin_inset Formula $k$
\end_inset

 catalogues, we obtain:
\end_layout

\begin_layout Standard
\begin_inset Formula 
\begin{equation}
P(D|H)=2^{k-1}\frac{\prod\sigma_{i}^{-2}}{\sum\sigma_{i}^{-2}}\exp\left\{ -\frac{\sum_{i<j}\psi_{ij}^{2}\sigma_{j}^{-2}\sigma_{i}^{-2}}{2\sum\sigma_{i}^{-2}}\right\} \label{eq:nwaylikelihood}
\end{equation}

\end_inset

For a given association with participating members from 
\begin_inset Formula $k$
\end_inset

 catalogues, the pairwise angular separation, 
\begin_inset Formula $\psi_{ij}$
\end_inset

, between catalogue 
\begin_inset Formula $i$
\end_inset

 and 
\begin_inset Formula $j$
\end_inset

 is judged with the relevant position uncertainties 
\begin_inset Formula $\sigma$
\end_inset

.
 The likelihood for the hypothesis where some catalogues do not participate
 in the association has the appropriate terms in the products and sums removed.
 Therefore, the likelihood is unity for the hypothesis that there is no
 counterpart in any of the catalogues.
\end_layout

\begin_layout Standard
In comparison to our method, the method of 
\begin_inset CommandInset citation
LatexCommand citet
key "Budavari2008"
literal "true"

\end_inset

 only compares two hypotheses for a association: either all sources belong
 to the same object (
\begin_inset Formula $H_{1}$
\end_inset

), or they are coincidentally aligned (
\begin_inset Formula $H_{0}$
\end_inset

).
 In this computation each hypothesis test is run in isolation, and relative
 match probabilities for a given source are not considered.
 For completeness, we also compute the posterior of this simpler model compariso
n:
\end_layout

\begin_layout Standard
\begin_inset Formula 
\begin{eqnarray}
\frac{P(H_{1}|D)}{P(H_{0}|D)} & \propto & \frac{P(H_{1})}{P(H_{0})}\times\frac{P(D|H_{1})}{P(D|H_{0})}\\
B & = & \frac{P(D|H_{1})}{P(D|H_{0})}\\
P(H_{1}|D) & = & \left[1+\frac{1-P(H_{1})}{B\cdot P(H_{1})}\right]^{-1}\label{eq:assocPost}
\end{eqnarray}

\end_inset

The output column 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

dist_bayesfactor
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 stores 
\begin_inset Formula $\log B$
\end_inset

, while the output column 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

dist_post
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 is the result of equation 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:assocPost"

\end_inset

.
 The output column 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_single
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 gives 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

dist_post
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 but modified if any additional information is specified (see Section 
\begin_inset CommandInset ref
LatexCommand ref
reference "sec:mag-priors"

\end_inset

).
 As mentioned several times in the literature, the 
\begin_inset CommandInset citation
LatexCommand citet
key "Budavari2008"
literal "true"

\end_inset

 approach does not include sources absent in some of the catalogues, while
 the formulae we develop below incorporate absent sources.
 This is similar in spirit to 
\begin_inset CommandInset citation
LatexCommand citealp
key "Pineau2016"
literal "true"

\end_inset

, although the statistical approach is different.
 We now go further and develop counterpart probabilities.
\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "subsec:Grouping-by-primary"

\end_inset

The first step in catalogue inference is whether the source has any counterpart
 (
\begin_inset Formula $p_{\mathrm{any}}$
\end_inset

).
 The posterior probabilities 
\begin_inset Formula $P(H|D)$
\end_inset

 are computed using Bayes theorem (eq.
 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:bayes"

\end_inset

) with the likelihood (eq.
 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:nwaylikelihood"

\end_inset

) and prior (eq.
 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:prior"

\end_inset

) appropriately adopted for the number of catalogues the particular association
 draws from.
 For each entry in the primary catalogue, the posteriors of all possible
 associations are normalised to unity, and 
\begin_inset Formula $P(H_{0}|D)$
\end_inset

, the posterior probability of the no-counterpart hypothesis, i.e., no catalogue
 participates, computed.
 From this we compute:
\end_layout

\begin_layout Standard
\begin_inset Formula 
\begin{equation}
p_{\mathrm{any}}=1-P(H_{0}|D)/\sum_{i}P(H_{i}|D)\label{eq:post-any}
\end{equation}

\end_inset

If 
\begin_inset Formula $p_{\mathrm{any}}$
\end_inset

 is low, this indicates that there is little evidence for any of the considered,
 combinatorically possible associations, except for the no-association case.
 The output column 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_any
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 is the result of equation 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:post-any"

\end_inset

.
 
\end_layout

\begin_layout Standard
If 
\begin_inset Formula $p_{\mathrm{any}}\approx1$
\end_inset

, there is strong evidence for at least one of the associations to another
 catalogue.
 To compute the relative posterior probabilities of the options, we re-normalize
 with the no-counterpart hypothesis, 
\begin_inset Formula $H_{0}$
\end_inset

, excluded:
\end_layout

\begin_layout Standard
\begin_inset Formula 
\begin{equation}
p_{i}=P(H_{i}|D)/\sum_{i>0}P(H_{i}|D)\label{eq:post-assoc}
\end{equation}

\end_inset

If a particular association has a high 
\begin_inset Formula $p_{i}$
\end_inset

, there is strong evidence that it is the true one, out of all present options.
 The output column 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

p_i
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 is the result of equation 
\begin_inset CommandInset ref
LatexCommand ref
reference "eq:post-assoc"

\end_inset

.
\end_layout

\begin_layout Standard
A 
\begin_inset Quotes eld
\end_inset

very secure
\begin_inset Quotes erd
\end_inset

 counterpart could be defined by the requirement 
\begin_inset Formula $p_{any}>95\%$
\end_inset

 and 
\begin_inset Formula $p_{i}>95\%$
\end_inset

, for example.
 However, it is useful to run simulations to understand the rate of false
 positives.
 Typically, much lower thresholds are acceptable.
\end_layout

\begin_layout Section
Magnitudes, Colors and other additional information
\end_layout

\begin_layout Standard
\begin_inset CommandInset label
LatexCommand label
name "sec:mag-priors"

\end_inset


\end_layout

\begin_layout Standard
Astronomical objects of various classes often show distinct color and magnitude
 distributions.
  Because most bright X-ray point-sources in deep images are also optically
 bright compared to generic sources, this information can be exploited.
 Previous works 
\begin_inset CommandInset citation
LatexCommand citep
before "e.g."
key "Brusa2005,Brusa2007"
literal "true"

\end_inset

 have modified the likelihood ratio coming from the angular distance 
\begin_inset Formula $f(r)$
\end_inset

 information (likelihood ratio method, 
\begin_inset CommandInset citation
LatexCommand citealp
key "SutherlandSaunders1992"
literal "true"

\end_inset

) by a factor:
\end_layout

\begin_layout Standard
\begin_inset Formula 
\begin{equation}
LR=\frac{q(m)}{n(m)}\times f(r)
\end{equation}

\end_inset

Here, 
\begin_inset Formula $q(m)$
\end_inset

 and 
\begin_inset Formula $n(m)$
\end_inset

 are associated with the magnitude distributions of source (e.g.
 X-ray sources) and background objects (e.g.
 stars, passive galaxies) respectively, but additionally contain sky density
 contributions.
\end_layout

\begin_layout Standard
This idea can be put on solid footing within the Bayesian framework.
 Here, two likelihoods are combined, by simply considering two independent
 observations, namely one for the positions, 
\begin_inset Formula $D_{\phi}$
\end_inset

, and one for the magnitudes 
\begin_inset Formula $D_{m}$
\end_inset

.
 The likelihood thus becomes
\end_layout

\begin_layout Standard
\begin_inset Formula 
\begin{eqnarray}
P(D|H) & = & P(D_{\phi}|H)\times P(D_{m}|H)\\
 & = & P(D_{\phi}|H)\times\frac{\bar{q}(m)}{\bar{n}(m)},
\end{eqnarray}

\end_inset

with 
\begin_inset Formula $\bar{q}(m)$
\end_inset

 and 
\begin_inset Formula $\bar{n}(m)$
\end_inset

 being the probability that a X-ray (target) source or a generic (field)
 source has magnitude 
\begin_inset Formula $m$
\end_inset

 respectively.
 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway
\end_layout

\end_inset

 stores the modifying factor, 
\begin_inset Formula $P(D_{m}|H)$
\end_inset

, in 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
code{
\end_layout

\end_inset

bias_*
\begin_inset ERT
status open

\begin_layout Plain Layout

}
\end_layout

\end_inset

 output columns, one for each column giving a magnitude, color, or other
 distribution.
 This modifying factor is however renormalized so that 
\begin_inset Formula $P(D_{m}|H)=\frac{\bar{q}(m)}{\bar{n}(m)}/\int\frac{\bar{q}(m')}{\bar{n}(m')}\bar{n}(m')dm'$
\end_inset

, which makes 
\family roman
\series medium
\shape up
\size normal
\emph off
\bar no
\strikeout off
\uuline off
\uwave off
\noun off
\color none

\begin_inset Formula $P(D|H)=P(D_{\phi}|H)$
\end_inset

 when 
\begin_inset Formula $m$
\end_inset

 is unknown.
 In that case, 
\begin_inset Formula $m$
\end_inset

 is marginalised over its distribution in the general population, i.e.
 
\begin_inset Formula $\int P(D_{m}|H)\,\bar{n}(m')\,dm$
\end_inset

.
 This has the benefit that when m is unknown, the modifying factor is unity
 and the probabilities remain unmodified.
\end_layout

\begin_layout Standard
For completeness, I mention the fully generalized case.
 This is attained when an arbitrary number of photometry bands are considered,
 each consisting of a magnitude measurement 
\begin_inset Formula $m$
\end_inset

 and measurement uncertainty 
\begin_inset Formula $\sigma_{m}$
\end_inset

:
\end_layout

\begin_layout Standard
\begin_inset Formula 
\begin{equation}
P(D_{m}|H)=\prod\frac{\int_{m}\bar{q}(m)\,p(m|D_{m})\,dm}{\int_{m}\bar{n}(m)\,p(m|D_{m})\,dm}
\end{equation}

\end_inset

Here, 
\begin_inset Formula $p(m|D_{m})$
\end_inset

 would refer to a Gaussian error distribution with mean 
\begin_inset Formula $m$
\end_inset

 and standard deviation 
\begin_inset Formula $\sigma_{m}$
\end_inset

.
 This is convolved with the distribution properties.
 Alternatively, 
\begin_inset Formula $p(m|D_{m})$
\end_inset

 can also consider upper limits.
 However, such options are not yet implemented in 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway
\end_layout

\end_inset

.
 Instead, we recommend removing magnitude values with large uncertainties
 (setting them to -99).
\end_layout

\begin_layout Section
Auto-calibration
\begin_inset CommandInset label
LatexCommand label
name "subsec:Auto-calibration"

\end_inset


\end_layout

\begin_layout Standard
The probability distributions 
\begin_inset Formula $\bar{n}(m)$
\end_inset

 and 
\begin_inset Formula $\bar{q}(m)$
\end_inset

 can be taken from other observations by computing the magnitude histograms
 of the overall population and the target sub-population (e.g.
 X-ray sources).
 
\end_layout

\begin_layout Standard
Under certain approximations and assumptions, these histograms can also
 be computed during the catalogue matching procedure while also being used
 for the weighting.
 One could perform the distance-based matching procedure laid out above,
 and compute a magnitude histogram of the secure counterparts as an approximatio
n for 
\begin_inset Formula $\bar{q}(m)$
\end_inset

 and a histogram of ruled out counterparts for 
\begin_inset Formula $\bar{n}(m)$
\end_inset

.
 While the weights 
\begin_inset Formula $\bar{q}(m)/\bar{n}(m)$
\end_inset

 may strongly influence the probabilities of the associations for a single
 object, the bulk of the associations will be dominated by distance-weighting.
 One may thus assume that the 
\begin_inset Formula $\bar{q}(m)$
\end_inset

 and 
\begin_inset Formula $\bar{n}(m)$
\end_inset

 are computed with and without applying the magnitude weighting are the
 same, which is true in practice.
 When differences are noticed, they will only strengthen 
\begin_inset Formula $\bar{q}(m)$
\end_inset

, and the procedure may be iterated.
\end_layout

\begin_layout Section
Implementation
\begin_inset CommandInset label
LatexCommand label
name "subsec:Implementation"

\end_inset


\end_layout

\begin_layout Standard
My implementation for matching 
\begin_inset Formula $n$
\end_inset

 catalogues is a Python program called 
\begin_inset ERT
status open

\begin_layout Plain Layout


\backslash
nway
\end_layout

\end_inset

.
 The input catalogues have to be in FITS format.
 Information about the (shared) sky coverage has to be provided to the program
 as well.
 The program proceeds in four steps.
\end_layout

\begin_layout Standard
First, possible associations are found.
 It is unfeasible and unnecessary to consider all theoretical possibilities
 (complexity 
\begin_inset Formula $O(\prod_{i=1}^{k}N_{i})$
\end_inset

), so the sky is split first to cluster nearby objects.
 For this, a hashing procedure puts each object into HEALPix bins 
\begin_inset CommandInset citation
LatexCommand citep
key "Gorski2005"
literal "true"

\end_inset

.
 The bin width 
\begin_inset Formula $w$
\end_inset

 is chosen so that any association of distance 
\begin_inset Formula $w$
\end_inset

 are improbable and negligible in practice, i.e.
 much larger than the largest positional error.
 An object with coordinates 
\begin_inset Formula $\phi,\,\theta$
\end_inset

 is placed in the bin corresponding to its coordinate, but also into its
 neighboring bins to avoid boundary effects.
 This is done for each catalogue separately.
 Then, in each bin, the Cartesian product across catalogues (every possible
 combination of sources) is computed.
 All associations are collected across the bins and filtered to be unique.
 The hashing procedure adds very low effort 
\begin_inset Formula $O(\sum_{i=1}^{k}N_{i})$
\end_inset

 while the Cartesian product is reduced drastically to 
\begin_inset Formula $O(N_{\text{bins}}\cdot\prod_{i=1}^{k}\frac{N_{i}}{N_{bins}})$
\end_inset

.
 All primary objects that have no associations past this step have 
\begin_inset Formula $P(\text{"any real association"}|D)=0$
\end_inset

.
\end_layout

\begin_layout Standard
The second step is the computation of posteriors using the angular distances
 between counterparts.
 The prior is also evaluated from the size of the catalogue and the effective
 coverage, as well as the user-supplied prior incompleteness factor.
 The posterior for each association based on the distances only is calculated.
 These posteriors have to be modified (
\begin_inset Quotes eld
\end_inset

correcting for unrelated associations
\begin_inset Quotes erd
\end_inset

), to consider associations unrelated to primary catalogue sources (described
 in the paper, 
\begin_inset CommandInset citation
LatexCommand cite
key "2018MNRAS.473.4937S"
literal "true"

\end_inset

, in the appendix section 
\begin_inset Quotes eld
\end_inset

Computing all possible matches
\begin_inset Quotes erd
\end_inset

).
\end_layout

\begin_layout Standard
In the third step the magnitudes are considered, and the posteriors modified.
 An arbitrary number of magnitude columns in the input catalogues can be
 specified.
 It is possible to use external magnitude histograms (e.g.
 for sparse matching with few objects) as well as computing the histograms
 from the data itself (see Section 
\begin_inset CommandInset ref
LatexCommand ref
reference "subsec:Auto-calibration"

\end_inset

).
 The breaks of the histogram bins are computed adaptively based on the empirical
 cumulative distribution found.
 Because the histogram bins are usually larger than the magnitude measurement
 uncertainty, the latter is currently not considered.
 The adaptive binning creates bin edges based on the number of objects,
 and is thus independent of the chosen scale (magnitudes, flux).
 Thus the method is not limited to magnitudes, but can be used for virtually
 any other known object property (colours, morphology, variability, etc.).
\end_layout

\begin_layout Standard
In the final step, associations are grouped by the object from the primary
 catalogue (here: X-ray source catalogue).
 The posteriors 
\begin_inset Formula $p_{\mathrm{any}}$
\end_inset

 and 
\begin_inset Formula $p_{i}$
\end_inset

 are computed.
 For the output catalogue a cut on the posterior probability (e.g.
 above 80%) can be applied, and all associations with their posterior probabilit
y are written to the output fits catalogue file.
\end_layout

\begin_layout Standard
\begin_inset CommandInset bibtex
LatexCommand bibtex
bibfiles "nway-references"
options "apalike2"

\end_inset


\end_layout

\end_body
\end_document
