/**
\page introduction Introduction

\imp is a library for solving a a wide variety of molecular structures and
dynamics using many different data sources. As a result, it provides a
great deal of flexibility. In order to best make the required decisions
about how to use \imp to solve a particular problem, it is useful to
understand the overall structure of \imp.

 -# \ref im1 "Theory"
 -# \ref i55 "Concepts"
 -# \ref i6 "Examples"
 -# \ref i125 "Modules"
 -# \ref i13 "C++ and Python"
 -# \ref conventions "Conventions"
 -# \ref incremental "Incremental scoring"
 -# \ref fixbug "Reporting bugs"
 -# \ref next "Where to go next"

\section im1 1. Theory

Structure and dynamics modeling in \imp proceeds in a five stage iterative process
\dotgraph{ \dot
 digraph intergrative_modeling {
    node [shape=record\, fontname= Helvetica\, fontsize=10]
      a [label="1. Gathering of experimental data"\, URL="\ref dataacq"];
      b [label="2. Choice of the representation\nscheme for the model"\, URL="\ref dataacq"];
      c [label="3. Encoding of the data as a\nscoring function on the\nrepresentation scheme"\, URL="\ref dataenc"];
      d [label="4. Sampling of the possible models\nin the representation scheme\nto try to find good models"\, URL="\ref sampling"];
      e [label="5. Analysis of the found models\nto determine if more work\nis needed"\, URL="\ref analysis"];
      a -> b [arrowhead="open"];
      b -> c [arrowhead="open"]
      c -> d [arrowhead="open"];
      d -> e [arrowhead="open"];
      e -> a [arrowhead="open"];
    }
\enddot
}
 
\imp provides a large number of functionality to facilitate this process. Links to representative classes are given for future reference.
-# \anchor dataacq <em> Data acquisition: </em> 
   we can't be much help here as we are computer people and don't know what to do with test tubes.
-# \anchor repsel <em> Representation selection: </em> representation in \imp is via a collection of
  entities called particles (IMP::Particle objects). Each particle can contain one or
  more of the following sets of data
   - \inlineimage{xyz.png, 50} Cartesian coordinates (IMP::core::XYZ)
   - \inlineimage{sphere.png, 50} sphere (IMP::core::XYZR)
   - \inlineimage{atom.png, 50} atom information such as type, element, mass (IMP::atom::Atom)
   - \inlineimage{residue.png, 50} residue type information (IMP::atom::Residue)
   - \inlineimage{chain.png, 50} chain IDs (IMP::atom::Chain)
   - \inlineimage{domain.png, 50} domain extents (IMP::atom::Domain)
   - \inlineimage{mass.png, 50} mass, in daltons (IMP::atom::Mass)
   - \inlineimage{charged.png, 50} charge (IMP::atom::Charged)
   - \inlineimage{hierarchy.png, 50} relationships between parts of molecules (IMP::atom::Hierarchy)
   - \inlineimage{bond.png, 50} bonds (IMP::atom::Bond and IMP::atom::Bonded)
   - \inlineimage{rigid.png, 50} rigid body coordinate frames (IMP::core::RigidBody)
   - \inlineimage{angle.png, 50} bond angle (IMP::atom::Angle)
   - \inlineimage{torsion.png, 50} bond angle (IMP::atom::TorsionAngle)
   - etc.
   .
  In addition, \imp can enforce relationships between particles:
   - \inlineimage{rigid.png, 50} all of the members of a rigid body move along with the rigid body (IMP::core::RigidBody, IMP::core::RigidMember)
   - \inlineimage{centroid.png, 50} a centroid particle has Cartesian coordinates computed from the centroid of another set (IMP::core::Centroid)
   - \inlineimage{spherecover.png, 50} a cover particle has a sphere containing another set of particles (IMP::core::Cover)
   .
  New types of representation can be easily added via a decorator mechanism, which is explained
  more below and on the IMP::Decorator page.\n\n
  Representations can loaded from a number of standard file types, for %example see
  IMP::atom::read_pdb() and IMP::atom::read_mol2().
-# \anchor dataenc <em>Encoding the data as a scoring function: </em>
  Proposed models are scored based on how well they match the
  data, with a low score meaning a closer fit than a high score. In
  \imp the scoring function is the sum terms, each of which is
  computed by an IMP::Restraint object. The scoring function terms can be
  based on things like
    - \inlineimage{distance.png, 50} how close a distance is to the measured value (IMP::core::DistanceRestraint, IMP::core::DistancePairScore, IMP::core::RigidBodyDistancePairScore, IMP::core::SphereDistancePairScore)
    - \inlineimage{em.png, 50} how well the model fits a density map (IMP::em::FitRestraint)
    - \inlineimage{volume.png, 50} how close the volume of a molecule is to the expected value (IMP::core::VolumeRestraint)
    - \inlineimage{ev.png, 50}  excluded volume (steric clash) (IMP::core::ExcludedVolumeRestraint)
    - \inlineimage{connectivity.png, 50} connectivity of a subcomplex (IMP::core::ConnectivityRestraint)
    - \inlineimage{saxs.png, 50} the fit of the SAXS cure of a complex to a measured one (IMP::saxs::Restraint)
    - \inlineimage{protlig.png, 50} statistical potentials (IMP::atom::ProteinLigandRestraint)
    - \inlineimage{angle.png, 50} torsion angles or bond angles (IMP::core::TorsionAngleRestraint, IMP::core::AngleRestraint)
    - \inlineimage{symmetry.png, 50} symmetry: (IMP::core::TransformedDistancePairScore)
    - etc.
    .
  Other terms can be formed by using IMP::SingletonScore, IMP::PairScore, IMP::TripletScore, IMP::QuadScore objects in conjuction with general purpose restraint creators. These allow large number of parts to be scored in similar ways more efficiently than creating many restraints. Examples using this include
    - symmetry by using IMP::core::TransformedDistancePairScore coupled with IMP::container::PairsRestraint
    - bond lengths using an IMP::atom::BondSingletonScore coupled with a IMP::container::SingletonsRestraint
    - etc.
    .
-# \anchor sampling <em> Sampling good conformations: </em> Once the scoring function has been designed you need to search for conformations of the model that
  have low scores (and therefore fit the data well). Sampling produces a set of
  conformations of the model, organized into an IMP::ConformationSet. Currently
  \imp provides two sampling protocols, IMP::core::MCCGSampler which uses a
  combination of IMP::core::MonteCarlo and IMP::core::ConjugateGradients with
  randomized starting conformations and IMP::domino::DominoOptimizer which uses
  a graph based inference algorithm. Sampling is an iterative process that tends to
  be structured as follows:
\dotgraph{ \dot
 digraph sampling {
    node [shape=record\, fontname= Helvetica\, fontsize=10]
      a [label="Sampler sets model to\ninitial\, random\nconfiguration"];
      b [label="Model gives constraints\nchance to update\nconfiguration"];
      c [label="Model asks restraints\nto compute score\n(and derivatives\, if desired)"];
      d [label="Model gives constraints\nchance to update derivatives\n(if desired)"];
      e [label="Sampler uses score\nand derivatives to\nmodify the configuration"];
      a -> b [arrowhead="open"];
      b -> c [arrowhead="open"]
      c -> d [arrowhead="open"];
      d -> e [arrowhead="open"];
      e -> b [arrowhead="open"];
    }
\enddot
}
-# \anchor analysis <em> Analysis of good conformations: </em> 
  Finally, one needs to analyze the set of conformations produced by
  sampling. \imp provides a variety of tools to help display the conformations,
  in IMP::display, and to cluster them, in IMP::statistics. Display capabilities
  include
  - export to Chimera via the IMP::display::ChimeraWriter
  - export to Pymol via the IMP::display::PymolWriter
  Clustering methods are currently based around k-means clustering
  (IMP::statistics::get_lloyds_kmeans()) and supports
  clustering of
  - configurations of the model using IMP::statistics::ConfigurationSetXYZEmbedding
  - arbitrary points using IMP::statistics::VectorDEmbedding
  - density maps using IMP::statistics::HighDensityEmbedding
  

Knowledge about the system being modeled enters the process at all
stages, but a few need extra note:

- The choice of representation must allow the final structure to be represented to
  the desired accuracy.

- A certain amount of knowledge is encoded as "constraints". These are
  choices made in choosing the representation scheme that force
  relationships between parts to always hold. For %example,
  representing part of the system as a rigid body, means that none of
  the distances or angles within the rigid body ever change.

- Other knowledge is encoded as restraints via scoring functions that
  penalize conformations which don't fit that bit of information.

- Knowledge can also be used to tune the sampling scheme by choosing
  starting configurations, how to perform the sampling etc.

Coming up with the right choices for representation, scoring and sampling for
a given system typically takes a few iterations and trial and error. \imp
provides tools to help monitor how things are performing.

- <em> Logging: </em> most actions in \imp can produce logged information to help understand
  what is going on. The amount of logging information produced can be controlled
  globally using the IMP.set_log_level() function and passing it one of the IMP::LogLevel
  values. In addition, restraints, samplers, constraints (and all objects which inherit
  from IMP::Object) have an internal log level that overrides the global one. To set
  that call the IMP::Object::set_log_level() function on that object. Setting the log
  level to IMP::VERBOSE will produce a huge amount of information during a typical
  sampling run. IMP::TERSE is generally better. Make sure to set it to at least
  IMP::WARNING to make sure that you don't miss any important warnings.

- <em> Usage checking: </em> \imp can perform a lot of checks that it is being used correctly
  as well as that it is behaving correctly. The checks being performed are controlled
  by the IMP::set_check_level() function call. Set the check level to IMP::USAGE to
  make sure that all parameters passed are correct. Set it to IMP::USAGE_AND_INTERNAL
  if you are developing new restraints or sampling protocols or are worried that
  \imp is malfunctioning.

- <em> IO: </em> \imp supports I/O to and from a variety of formats. To preserve the maximum amount
  of information, one can use the IMP::read_model() and IMP::write_model() methods to
  save and load a whole model to a human (and machine) readable file. See also the
  IMP::display module for geometry output and IMP::atom for biological formats.

\section i55 2. Concepts
As has already been hinted at, \imp is organized around a number of core concepts.
Representation is handled via a collection of IMP::Particle objects. Each has a
set of arbitrary attributes (such as an x coordinate or a mass). In order to make
particles more friendly, we provide decorators which, guess what, decorate, an existing
particle to provide a higher level interface to manipulate the attributes of a particle.
See the IMP::Decorator page for more details.

\imp provides containers in order to aid managing sets of particles. These inherit from
IMP::Container (notice that IMP::Particle objects are containers and can contain lists
of particles). A container could
be as simple as an IMP::container::ListSingletonContainer which simply stores a list
of particles. However, it could also be more involved, such as the IMP::container::ClosePairContainer which keeps track of all pairs of particles which close to one another in space. It can
be used to implemented non-bonded operations for example.

Scoring is handled by a collection of IMP::Restraint objects. Each of these keeps a list
of particles and scores those particles based on how well they fit some sort of data.

The IMP::Model manages the set of all particles in the representation along with
the set of all restraints scoring them and constraints acting on them. It provides
one central function, IMP::Model::evaluate() which computes the score of the current
conformation.

One final representation concept is that of
a constraint. These are implemented as IMP::Constraint objects. They maintain
some hard invariant of the representation. Examples include, keeping a rigid body
rigid, or ensuring that the IMP::container::ClosePairContainer really contains all
close pairs. Constraints are updated as part of the IMP::Model::evaluate().
<em> This means that the constraint does not necessarily hold except during
score evaluation.</em> In order to ensure that all constraints hold, call
IMP::Model::evaluate() before inspecting the particles.

On the sampling side, there are two main concepts, that of an optimizer and that of
a sampler. An optimizer (IMP::Optimizer), takes the current conformation of the
IMP::Model and modifies it (typically in an attempt to make it score better). A
sampler uses variety of optimizers and other methods to perform a non-local
search for good scoring conformations, which are then stored as part of an
IMP::ConfigurationSet.


\section i6 3. Examples
The following examples give some idea of the basics of using \imp. They are all are in Python,
but the C++ code is nearly the same.

Each module has an examples page linked from its main page.

\subsection i7 Creating some particles
The function creates a bunch of particles and uses the IMP::core::XYZR decorator to
given them random coordinates and a radius of 1.
\htmlinclude setup.py.html

\subsection i7 Restrain some distances

Once the particles are created, we have to add some restraints. To do this, you must choose
which particles to restraint and then how to restrain them. Given that you create a restraint,
initializing it with the chosen particles and then add it to the model.

\htmlinclude pair_restraint.py.html


\subsection i8 Preventing collisions
The IMP::container::ClosePairsContainer maintains a list of all pairs of particles that are closer than a certain distance. The IMP::core::HarmonicLowerBound forces the spheres apart.
\htmlinclude excluded_volume.py.html


\subsection i9 Restraining bonds

Load a protein and restrain all the bonds to have the correct length. Bond angles is a bit trickier at the moment.

\htmlinclude load_protein_restrain_bonds.py.html






\subsection i10 Sampling and analysis
Once we have set up our restraints, we can run a sampler to compute some good
conformations. Our basic sampler is the IMP::core::MCCGSampler which uses
a combination of Monte Carlo and conjugate gradients to find conformations.
It then returns an object which allows one to load the saved conformations
for analysis.
\htmlinclude basic_optimization.py.html

\subsection i11 Writing a simple restraint
See IMP::example::ExampleRestraint.

\section i125 4. Modules

Functionality in \imp is grouped into modules, each with its own
namespace (in C++) or package (in Python). For %example, the functionality
for IMP::core can be found like
\code
IMP::core::XYZ(p)\endcode
in C++ and
\verbatim
IMP.core.XYZ(p)\endverbatim
in Python.

A module contains classes,
methods and data which are related and controlled by a set of authors. The names
of the authors, the license for the module, its version and an overview of the
module can be found on the module main page (eg IMP::example).
See the "Modules" tab above for a complete list of modules in this version of \imp.

Modules are either grouped based on types of experimental data (eg IMP::em) or based
on shared functionality (IMP::core or IMP::container).


\section i13 5. C++ vs Python
\anchor cpppythondifferences
\imp can be used from both C++ and Python. We recommend that you:
- use Python to put \imp classes together to handle your data and resulting structures
- write new \imp classes in C++

If you are new to programming you should check out a general python 
introduction such as
\external{docs.python.org/tutorial/introduction.html, the official introduction to Python} and
\external{www.rexx.com/~dkuhlman/python_101/python_101.html, Python
101}. Users who have
programmed but are not familiar with Python should take a look at
\external{diveintopython.org/toc/index.html, Dive into Python},
especially chapters 1-6, and 15-18.

While effort has been made to ensure that the interfaces are the same
between the two languages, a number of differences remain due to
differences in the languages and limitations of the program used to
generate the connection between the two languages. Key differences are

- Python does not support templates and so template classes (eg
  IMP::algebra::BoundingBoxD, IMP::VectorOfRefCounted) cannot be
  directly exposed in python. Instead, specific versions of the
  classes are exported (IMP::algebra::BoundingBoxD<3> and
  IMP::Particles, respectively). New versions can be easily added, so
  feel free to request them when desired.
- Iterators on C++ containers do not translate easily into Python. As a
  result, for every iterator generating pair \c foos_begin(),
  \c foos_end(), we provide a method \c get_foos() which can be used
  with python \c foreach loops.
- Macros such as IMP_RESTRAINT(), IMP_LOG(), IMP_USAGE_CHECK() are not
  available in Python. While we could, conceivably, provide python
  function equivalents, we do not.
- Oddly, the Python side is much stricter about converting between
  different types. In C++ you can call a function that takes a \c
  Particle* with a \c Decorator and the decorator will automatically
  be converted. It will not on the python side. Similarly for
  converting between \c ParticlesTemp and \c Particles.
- All \imp exceptions are exposed as identical Python exception
  classes. The class hierarchy is similar (e.g. all exceptions derive
  from IMP::Exception, so "except IMP.Exception" will catch all IMP
  exceptions), except for convenience some generic \imp exceptions also
  derive from their standard Python equivalents
  (e.g. IMP.IndexException derives from the standard Python IndexError
  as well as IMP::Exception). Thus, an IMP::IndexException could be
  caught in Python most specifically with "except IMP.IndexException"
  but also with "except IMP.Exception" or "except IndexError".
- All objects in python are reference counted so that they are cleaned
  up when they are no longer in use. \imp also uses reference counting
  on the C++ side so that memory managment works naturally across the
  language barrier. See IMP::RefCounted for a detailed description of
  how to do \imp reference counting in C++.


\section conventions 6. Conventions

    To ensure consistency and ease of use, certain conventions should be
    adhered to when writing code using or in \imp.

    \subsection physical Measurements
    Unless there is a good reason, the following units are to be used
    - angstrom for all distances
    - \f$ \frac{\operatorname{kcal}}{\operatorname{mol} \AA}\f$ for forces/derivatives
    - \f$\frac{\operatorname{kcal}}{\operatorname{mol}}\f$ for energies
    - radians for angles. All angles are counterclockwise.
    - all charges are in units of the elementary charge

    Anything that breaks from these conventions must be labeled
    clearly and accompanied by an explaination of why the normal units
    could not be used.

    \subsection storage Passing and storing data

    - 3D points and vectors are stored and passed using
      IMP::algebra::VectorD objects.

    - 3D rotations are stored and passed using
      IMP::algebra::Rotation3D objects.

    - Likewise for spheres (IMP::algebra::SphereD), segments
      (IMP::algebra::Segment3D) etc.

    - Collections of object \c Name are passed using the type \c Names. For
    %example, a bunch of IMP::algebra::Vector3D objects are passed using
    a IMP::algebra::Vector3Ds type, and a bunch of IMP::Restraint
    objects is passed using IMP::Restraints (or, equivalently
    IMP::RestraintsTemp).

    - Classes and methods use \imp exceptions to report errors. See
    IMP::Exception for a list of existing exceptions. These C++
    exceptions are mapped onto the normal python exception types.

    \anchor values
    \subsection objects Values and Objects
    As is conventional in C++, \imp classes are divided into two types
    - \e value \e classes which are be passed, stored, and returned by value
      (or, for speed, \c const&). Examples include IMP::algebra::VectorD,
      collections such as IMP::Restraints or IMP::RestraintsTemp, or
      decorators, such as IMP::core::XYZ. In fact, in \imp, anything
      that does not inherit from IMP::RefCounted is a value class.

    - \e object \e classes which are passed and returned via pointers and
      stored using reference counted pointers (eg IMP::Pointer). In
      \imp, these classes all inherit from IMP::Object. For %example always
      do things like this in C++:
      \code
      IMP::Pointer<IMP::Model> m= new IMP::Model();
      IMP_NEW(Model, m, ()); // a macro which expands to the above
      \endcode
      \anchor tempornot Since reference counting can be expensive,
      it can be useful to pass, return or store a non-reference
      counted list of objects (or decorators). This should only
      be done when it is known to be safe. If you can't figure out
      that it is, don't do it. If it is safe, pass a \c NamesTemp
      instead of a \c Names.

    Python does not have this distinction.

    A few classes in \imp are designed for fast, low level use. Their default
    constructor leaves them in an unspecified state. This is similar to
    the built in types in C++ (\c int, \c double). For %example
      \code
      IMP::algebra::VectorD<3> v; // the vector has unknown coordinates
      std::cout << v << std::endl; // illegal
      v= IMP::algebra::VectorD<3>(0,1,2); // now we can use v
      \endcode

    Unless the documentation says otherwise, all \e value
    \e class object in \imp can be compared with other equivalent objects
    based on their contents. \e Object \e class objects allow checking of
    equality to see if they are the same object (not whether two have the
    same state). In C++, this is done by comparing the pointers.
   \subsection standardmethods Standard Methods
    All objects should have a \c const method \c show(std::ostream&), which
    writes some basic information about the object to the supplied stream.
    In addition, on the C++ side, all objects support standard output
    to stream via \c <<. In addition, all objects support \c __str__ in
    python so that they can be printed and displayed.

   \subsection names Names in IMP
    - Class names are in \c CamelCase, for %example \c class \c SpecialVector'
    - For each type of object in \imp, \c Name, there is a type \c Names
      which is used to pass a list of objects of type \c Name. \c Names
      look like an \c std::vector in C++ or a \c list in Python. Sometimes,
      for efficiency, a \c NamesTemp is passed instead
      (see \ref tempornot "when to use Temp values" for the reason). \c Names
      will be converted into \c NamesTemp without cost, so the distinction
      should not matter for the caller.
    - method names and variables are \c separated_by_underscores, for
      %example \c void \c SpecialVector::add_constant(int the_constant)'
    - member methods that change a value begin with \c set_
    - member methods or function which create or return a \c value object or which
      return an existing \c object \c class object begin with \c get_. No arguments
      of such functions are modified.
    - methods or functions which create a new IMP::Object \c class object start
      with \c create_.
    - functions with names starting with other verbs have more complicated effects. For
      %example IMP::core::transform() changes the first argument based on the second.
      IMP::Restraint::evaluate() computes a score as well as (optionally) adding to
      Particle derivatives.
    - all preprocessor symbols (things created by \c \#define) begin with \c %IMP_
    - Abbreviations are not used in names except when the abbreviation is more
      common than the unabreviated name.

\section incremental 7. Incremental Scoring
    Scoring in \imp can be performed in two different ways....

    Incremental scoring  works as follows:

    To set it up call IMP::Model::set_is_incremental() with the value \c true. This
    -# calls regular evaluate on all incremental restraints
    -# a shadow particle is added to each particle. The shadow particles have all the same attributes. It is accessed using IMP::Particle::get_prechange_particle()
    -# saves copies of their derivatives to the shadow particles

    When evaluate is called during optimization
    -# the derivatives are cleared on all the particles (but not the shadow particles)
    -# incremental restraints are evaluated. They need to make sure that the the change in the sum of the particle and shadow particle derivatives is equal to the change in derivatives and that the actual score is returned.
    -# derivatives are added to the shadow derivatives and then cleared
    -# the non-incremental restraints are then evaluated
    -# the derivatives of the shadow particles are added to the particle derivatives
    -# after scoring, all the particles are marked as clean and shadow particles are updated to reflect the current attributes of the particles

    A IMP::Restraint is an incremental restraint if IMP::Restraint::get_is_incremental() returns true. For such restraints,
    IMP::Restraint::incremental_evaluate() is called instead of IMP::Restraint::evaluate().

    Whenever a particle is changed is marked as dirty, so that IMP::Particle::get_is_changed() returns true.

    A (perhaps partial) list of classes which benefit from incremental evaluation is:
    - IMP::core::IncrementalBallMover
    - all the optimizers/samplers

\section fixbug 8. Reporting bugs

While we strive for perfection, we, lamentably, slip up from time to
time. If you find a bug in \imp, please report it on the
\salilabssl{imp/bugs/,IMP bug tracker}. This will ensure it does not get
lost.  The best way to report a bug is to provide a short script file
that demonstrates the problem.



\section next 9. Where to go next
Instructions on how to build and install \imp can be found in
the \ref installation "installation instructions".

There are a few areas of core functionality that have already been mentioned.
- IMP::Decorator
- \ref log "Logging Support"
- \ref assert "Error Reporting Facilities"

Then look through the examples which can be found linked from the page of each module.
 
There are a variety of useful base classes which are used to provide
most functionality. They are:
    - IMP::Model
    - IMP::Particle
    - IMP::Decorator
    - IMP::Restraint
    - IMP::Constraint
    - IMP::Sampler
    - IMP::Optimizer
    - IMP::SingletonContainer, IMP::PairContainer
    - IMP::SingletonScore, IMP::PairScore, IMP::TripletScore
    - IMP::SingletonModifier, IMP::PairModifier

There are a few blocks of functionality that cut across modules. They
include
- \ref hierarchy "Code to create and manipulate hierarchies"
- \ref bond "Code to create and manipulate bonds"

When programming with \imp, one of the more useful pages is the
\internal{namespaces.html, modules list}.

For general help, you can use the \impusers mailing list.

*/