# Jscatter’s documentation

The aim of Jscatter is treatment of experimental data and models:

- Reading and analyzing experimental data with associated attributes as temperature, wavevector, comment, ….
- Multidimensional fitting taking attributes into account.
- Providing useful models for neutron and X-ray scattering form factors, structure factors and dynamic models (quasi elastic neutron scattering) and other topics.
- Simplified plotting with paper ready quality (preferred in xmgrace).
- Easy model building for non programmers.
- Python scripts to document data evaluation and modelling.

Try Jscatter live at . Cite Jscatter by .

Main concept

- Link data from experiment, analytical model or simulation with attributes as .temperature, .wavevector, .pressure,…
- Methods for fitting, filter, merging,… using the attributes by name.
- Provide an extensible library with common theories for fitting of physical models.

1. Data organisation

> Multiple measurements are stored in a `dataList` (subclass of list) containing `dataArray` ´s (subclass of numpy ndarray) for each measurement. Both allow attributes to contain additional information of the measurement.

> Thus dataList represents e.g. a temperature series (as dataList) with measurements (dataArray) as list elements.

> Special attributes are .X,.Y,.eY,…- for convenience and easy reading. Full numpy ndarray functionality is preserved.

2. Read/Write data

> The intention is to read everything (with comments) from a file to use it later if needed. Multiple measurement files can be read at once and then filtered according to attributes to get subsets.

> A file may consist of multiple sets of data with optional attributes or comments in between. Data are a matrix like values in a file. Attribute lines have a name in front. Everything else is a comment and might be used later. Thus the first two words (separated by whitespace) decide about assignment of a line:

> - string + value -> attribute with attribute name + list of values
> - value + value -> data line as sequence of numbers
> - string + string -> comment
> - single words -> comment
> - string+@unique_name-> link to other dataArray with a unique_name

> Even complex ASCII files can be read with a few changes given as options. The ASCII file is still human readable and can be edited. New attributes can be generated from content of the comments if not detected automatically (see Reading ASCII files).

3. Fitting

> Multidimensional, attribute dependent fitting (least square Levenberg-Marquardt, differential evolution, …from scipy.optimize).

> Attributes are used automatically as fixed fit parameters.

> Simulation with changed parameters (e.g. to observe change within error limits).

> See `fit()` for detailed description and examples in 1D fits with attributes or 2D fitting.

4. Plotting

> The aim is to provide one line plotting commands to allow a fast view on data, with the possibility to pretty up the plots.

> - We use an adaption of Xmgrace for 2D plots (a wrapper; see GracePlot) as it allows interactive publication ready output in high quality for 2D plots and is much faster than matplotlib.

>   The figure is stored as ASCII file (.agr) including data points and not as non-editable image as jpg/pdf… This allows a later change of the plot layout without recalculation, because data are stored as data and not as image. Imagine the boss/reviewer asking for a change of colors/symbol size.

> - A small matplotlib interface is provided and matplotlib can be used as it is (e.g. for 3D plots).

> - Still any other plotting package can be used.

5. Model Library

> By intention the user should write own models or modify existing ones to combine different contributions (to include e.g. a background, instrument resolution, …).

> New models dont`t need to be registered or placed/compiled into Jscatter. Models can be defined as lambda function or normal functions within a script or in interactive session of (I)python. Or you write your own local module as collection of your private functions to import. See How to build simple models and How to build a more complex model .

> The model library contains general purpose routines e.g. for vectorized quadrature (formel) or specialised models for scattering in formfactor (ff), structurefactor (sf) and dynamic. Models contain model parameters as attributes for later access. The model library can also be used for other purposes and may be extended by users need.

> Contribution by new models is welcome. Please give a documentation, reference to relevant publication and authorship as in the provided models.

Some special functions:

- `scatteringLengthDensityCalc()` -> Electron density, coh and inc neutron scattering length, mass
- `waterdensity()` -> Density of water (H2O/D2O) with inorganic substances
- `sedimentationProfile()` -> The Lamm equation of sedimenting particles
- `RMSA()` -> Rescaled MSA structure factor for dilute charged colloidal dispersions
- `hydrodynamicFunct()` -> Hydrodynamic function from hydrodynamic pair interaction
- `multiShellSphere()` -> Formfactor of multi shell spherical particles
- `multiShellCylinder()` -> Formfactor of multi shell cylinder particles with caps
- `orientedCloudScattering()` -> 2D scattering of an oriented cloud of scatterers
- `finiteZimm()` -> Zimm model with internal friction -> intermediate scattering function
- `diffusionHarmonicPotential()` -> Diffusion in harmonic potential-> intermediate scattering function
- `smear()` -> Smearing for SANS (Pedersen), SAXS (line collimation) or by explicit Gaussian
- `desmear()` -> Desmearing according to the Lake algorithm for the above
- `waterXrayScattering()` -> Absolute scattering of water with components (salt, buffer)

How to use Jscatter or see Examples and Beginners Guide / Help or try Jscatter live at .

    # import jscatter and numpy
    import numpy as np
    import jscatter as js

    # read the data (16 sets) with attributes as q, Dtrans .... into dataList
    i5 = js.dL(js.examples.datapath + '/iqt_1hho.dat')

    # define a model for the fit
    diffusion = lambda A, D, t, elastic, wavevector=0: A * np.exp(-wavevector ** 2 * D * t) + elastic
    # do the fit
    i5.fit(model=diffusion,  # the fit function
           freepar={'D': [0.08], 'A': 0.98},  # start parameters, "[]" -> independent fit
           fixpar={'elastic': 0.0},  # fixed parameters
           mapNames={'t': 'X', 'wavevector': 'q'})  # map names from the model to names from the data
    # single valued start parameters are the same for all dataArrays
    # list start parameters indicate independent fitting for datasets
    # the command line shows progress and the final result, which is found in .lastfit
    i5.showlastErrPlot(yscale='l')  # opens plot with residuals

    # open a plot with fixed size and plot
    p = js.grace(1.2, 0.8)
    # plot the data with Q values in legend as symbols
    p.plot(i5, symbol=[-1, 0.4, -1], legend='Q=$q')
    # plot fit results in lastfit as lines without symbol or legend
    p.plot(i5.lastfit, symbol=0, line=[1, 1, -1])

    # pretty up if needed
    p.yaxis(min=0.02, max=1.1, scale='log', charsize=1.5, label='I(Q,t)/I(Q,0)')
    p.xaxis(min=0, max=130, charsize=1.5, label='t / ns')
    p.legend(x=110, y=0.9, charsize=1)
    p.title('I(Q,t) as measured by Neutron Spinecho Spectroscopy', size=1.3)
    p.text('for diffusion a single exp. decay', x=60, y=0.35, rot=360 - 20, color=4)
    p.text(r'f(t)=A*e\S-Q\S2\N\SDt', x=100, y=0.025, rot=0, charsize=1.5)

    if 0:  # optional; save in different formats
        p.save('DiffusionFit.agr')
        p.save('DiffusionFit.jpg')

Shortcuts:

    import jscatter as js
    js.showDoc()                  # Show html documentation in browser
    exampledA=js.dA('test.dat')   # shortcut to create dataArray from file
    exampledL=js.dL('test.dat')   # shortcut to create dataList from file
    p=js.grace()                  # create plot in XmGrace
    p=js.mplot()                  # create plot in matplotlib
    p.plot(exampledL)             # plot the read dataList
    js.usempl(True)               # use matplotlib for residual plots in fits

If not otherwise stated in the files:
written by Ralf Biehl at the Forschungszentrum Jülich ,
Jülich Center for Neutron Science 1 and Institute of Complex Systems 1
Jscatter is a program to read, analyse and plot data
Copyright (C) 2015-2019 Ralf Biehl
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.

# Jscatter package contents

- 1. Beginners Guide / Help
  - 1.1. What are dataArray/dataList
  - 1.2. First basic examples
    - 1.2.1. A parabolic fit
    - 1.2.2. A bit more complex data
  - 1.3. Reading ASCII files
  - 1.4. Creating from numpy arrays
  - 1.5. Manipulating dataArray/dataList
  - 1.6. Indexing dataArray/dataList and reducing
  - 1.7. Fitting experimental data
  - 1.8. Why Fits may fail
  - 1.9. Plot experimental data and fit result
  - 1.10. Save data and fit results
  - 1.11. Additional stuff
- 2. dataArray
  - 2.1. dataArray Class
  - 2.2. Attribute Methods
  - 2.3. Fit Methods
  - 2.4. Housekeeping Methods
  - 2.5. Convenience
- 3. dataList
  - 3.1. dataList Class
  - 3.2. Attribute Methods
  - 3.3. Fit Methods
  - 3.4. Housekeeping Methods
- 4. formel
  - 4.1. Functions
  - 4.2. Quadrature
  - 4.3. Centrifugation
  - 4.4. NMR
  - 4.5. Material Data
  - 4.6. other Stuff
  - 4.7. Constants and Tables
- 5. smallanglescattering (sas)
  - 5.1. Smear/desmear 1D
  - 5.2. 2D sasImage
    - 5.2.1. sasImage methods
  - 5.3. 2D sasImage convenience
  - 5.4. Housekeeping
- 6. formfactor (ff)
  - 6.1. Form Factors
  - 6.2. Cloud of scatterers
  - 6.3. Distribution
- 7. structurefactor (sf)
  - 7.1. Structure Factors
  - 7.2. Hydrodynamics
  - 7.3. Pair Correlation
  - 7.4. Lattice
- 8. dynamic
  - 8.1. Transform between domains
  - 8.2. Time domain
  - 8.3. Frequency domain
- 9. dls
- 10. parallel
- 11. Plotting in XmGrace
  - 11.1. GracePlot
  - 11.2. GraceGraph
  - 11.3. Helper Classes
- 12. mpl
  - 12.1. Plot
  - 12.2. Figures
  - 12.3. Axes
- 13. Examples
  - 13.1. In a hurry and short
  - 13.2. How to build simple models
  - 13.3. How to build a more complex model
  - 13.4. 1D fits with attributes
  - 13.5. 2D fitting
  - 13.6. Simple diffusion fit of not so simple diffusion case
  - 13.7. How to smooth Xray data and make an inset in the plot
  - 13.8. Analyse SAS data
  - 13.9. How to fit SANS data including the resolution for different detector distances
  - 13.10. Smearing and desmearing of SAX and SANS data
  - 13.11. A long example for diffusion and how to analyze step by step
  - 13.12. Sedimentation of two particle sizes and resulting scattering: a Simulation
  - 13.13. Create a stacked chart of some curves
  - 13.14. A comparison of different dynamic models in frequency domain
  - 13.15. Protein incoherent scattering in frequency domain
  - 13.16. Fitting a multiShellcylinder in various ways
  - 13.17. Hydrodynamic function
  - 13.18. Multilamellar Vesicles
  - 13.19. 2D oriented scattering
  - 13.20. Fitting the 2D scattering of a lattice
  - 13.21. A nano cube build of different lattices
  - 13.22. Using cloudscattering as fit model
- 14. Extending/Contributing/Fortran
- 15. Tips
- 16. Intention and Remarks
- 17. Installation
  - 17.1. Dependencies
  - 17.2. Pip installation/upgrade
  - 17.3. Linux
  - 17.4. MacOs
  - 17.5. Windows
    - 17.5.1. Windows Subsystem for Linux with Ubuntu way
    - 17.5.2. Anaconda way
  - 17.6. Jupyter Notebook
  - 17.7. Testing
  - 17.8. Troubleshooting and tips
- 18. Citing Jscatter
- 19. Changelog

# Installation

## Dependencies

- numpy, scipy -> Mandatory, automatically installed by pip
- Pillow -> Mandatory, for reading of SAXS images, automatic install by pip
- matplotlib -> Mandatory, for 3D plots and on Windows
- Ipython -> Optional, for convenience as a powerful python shell
- gfortran -> Optional, without some functions dont work or use a slower python version
- xmgrace -> Optional, preferred plotting on Unix like (use matplotlib on Windows)

Installation of gfortran/xmgrace may need root privileges. Use “sudo” on Linux and MacOS if needed.

## Pip installation/upgrade

(use pip2 or pip3 if NOT your default Python should be used)

    sudo pip install jscatter

As user in home directory (pip default installation directory is in ~/.local/). No sudo needed, only user privileges, you don’t need the admin to install/update:

    pip install jscatter --user

from a local repository (development versions):

    pip install jscatter --user --upgrade --pre --find-links /where/the/file/is/saved

    options
    --user       : Install in user directory (folder defined by PYTHONUSERBASE or the default ~/.local)
    --find-links : look in the given path for package links e.g development releases
    --upgrade    : to install upgrades also for dependencies
    --pre        : to install also development versions

## Linux

- Ubuntu, all Debian related

      sudo apt-get install gfortran grace python-matplotlib  # or python3-matplotlib
      sudo pip install ipython
      sudo pip install jscatter

- CentOs, Suse, Fedora … do same as above but with yum/zypper…

- Manjaro Linux (yaourt asks for permission as root or prepend sudo as above)

      # install gfortran
      yaourt gcc-fortran
      # install xmgrace (only found in AUR), fonts are needed for the interface, fonts are loaded after restart
      yaourt xorg-fonts-75 xorg-fonts-100 grace-openmotif python-matplotlib
      # tk might be missing for tkinter as matplotlib backend
      sudo pacman -S tk
      pip install ipython
      pip install jscatter

- CONTIN in DLS module (Only if needed).

  See DLS module documentation for details how to get and compile the original fortran code.

## MacOs

Install Homebrew first as given on their web page (see Homebrew)

    # install XQuartz from homebrew or from the AppStore
    sudo brew cask install xquartz
    # install xmgrace and gfortran
    sudo brew install grace gfortran matplotlib
    # then use pip
    sudo pip install ipython
    sudo pip install jscatter

## Windows

### Windows Subsystem for Linux with Ubuntu way

A new way is based on Windows Subsystem for Linux (WSL), which “lets developers run GNU/Linux environment – including most command-line tools, utilities, and applications – directly on Windows, unmodified, without the overhead of a virtual machine.” (See WSL)

This allows to use any Linux application including XmGrace or Linux editors. You work on the same filesystem as your Windows user account without the need of syncing folders or using a shared folder from a virtual machine. Up to now i didnt test gfortran.

The basic procedure is to activate WSL, install Linux and Xserver, add your needed Linux software in the usual way.

- Install WSL

  See https://docs.microsoft.com/de-de/windows/wsl/install-win10

  or in Powershell as administrator

      Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

- Then install a linux distribution of choice from Windows store (tested with Ubuntu 18.04). (The install is per user, so do it under your user account, not as administrator)

  Open a terminal, start bash and set user and password for Linux. (See https://docs.microsoft.com/en-us/windows/wsl/initialize-distro )

- For graphical applications: Install Xserver VcXsrv (or an other Xserver).

  See e.g. https://seanthegeek.net/234/graphical-linux-applications-bash-ubuntu-windows/ . Download from https://sourceforge.net/projects/vcxsrv/ . Install (start always manually or add to Windows autostart).

- Configure bash to use the local X server

  Add this to .bashrc

      export DISPLAY=localhost:0.0

  or run this inside bash

      echo "export DISPLAY=localhost:0.0" >> ~/.bashrc

  Restart bash or load again .bashrc `source ~/.bashrc`

- Install xmgrace, python3 and Jscatter in Linux subsystem

  open terminal, start bash

      sudo apt-get update && upgrade                        # upgrade linux
      sudo apt-get install python3 python3-pip ipython3     # python, ipython and pip3
      sudo apt-get install numpy python3-matplotlib xmgrace
      pip3 install jscatter                                 # jscatter, scipy, Pillow...

> Run test and example.

### Anaconda way

Anaconda is a python distribution as alternative with numpy, scipy, matplotlib, Ipython preinstalled. Need of sudo depends on how Anaconda was installed (root or user). Maybe the matplotlib backend needs to be configured on Windows to work properly.

And there was more to adjust, when i stopped waisting my time. In my testcase it was a pain, but test and examples work (no Xmgrace, no fortran).

I strongly advise to use Linux in a VirtualMachine as it is easier to install and use.

    # install jscatter on working anaconda environment
    pip install jscatter

## Jupyter Notebook

Jscatter works on Jupyter Notebooks.

To install jscatter on a server Jupyter installation (Jscatter not preinstalled) prepend this on your script

    import sys,site
    # install jscatter as user in the current Jupyter kernel
    !{sys.executable} -m pip install jscatter --user
    # append user install dir to path
    sys.path.append(site.USER_BASE)

There is some trouble with inline plots (which are not interactive and cannot be updated) dependent on the installed backend. This is related to the used matplotlib backend.

Use this (before importing anything else) to get interactive plots inline.

    %matplotlib notebook

Then import Jscatter.

    import jscatter as js

    js.usempl(True)  # use matplotlib
    js.usempl(False) # default, use grace on your computer with xmgrace in external window.

If you work over http on a server (usual no display of Xwindows applications) use the same to switch to matplotlib.

## Testing

You can test basic functionality of jscatter after installation:

    import jscatter as js
    js.test.doTest()
    #basic graphics and fitting
    js.examples.runExample('example_SinusoidalFitting.py')

The Example shows :
  - 3 sine fit plots with one sine
  - a fit plot with 5 sine curves fitted simultaneous
  - a simple plot with 5 points ( phase against Amplitude of the 5 sines)

During development:

    python setup.py test

## Troubleshooting and tips

If xmgrace is not found by jscatter the path to the executable may be not on your PATH variable. Check this by calling xmgrace in a shell. Change your PATH in your .bashrc by adding:

    export PATH=/path/to/xmgrace:$(PATH) )

To open .agr files by klicking add a new file association e.g. to KDE.

  In SystemSettings/FileAssociations add a new type xmgrace. Inside this add FilenamePatterns ‘*.agr’ and similar.

  > In ‘ApplicationPreferenceOrder’ add xmgrace and edit this new application :
  >   In General : edit name to “xmgrace” (keep it). In Application : edit name to “xmgracefree” and command to “xmgrace -free”. This will open files in free floating size format.

  > In ‘ApplicationPreferenceOrder’ add again application xmgrace (no changes) The second opens files in fixed size format (no ‘-free’).

# Indices and tables

- Index
- Module Index
- Search Page