The Scan Emit Loop
==================

Core in *data-migrator* is the declarative definition of the target model,
indeed in a Django-esc way. Columns of the target table are defined as fields
and each field has many settings. The Field is a definition of what to perform
when scanning, transforming and emitting the record. Output is abstracted to
an extensible set of output writers, called emitters. The whole is
controlled with a standard transformer engine.

The scan-emit loop is the basis the *data-migrator*. Once everything is setup,
by default the transformer will read stdin and send every CSV row to the model
for scanning. Out of the box the fields define a scan loop:

#. **select** the specified column from the row.
#. **nullable** test if not allowed and replace by None.
#. **validate** the input (if validator is provided).
#. **parse** the input (if parser is provided).
#. **store** as native python value (aka NULL=>None).

Once all fields are parsed, the resulting object can be checked for `None`
or uniqueness. It can be dropped or the filter can fail because of violations.
This are all declarative settings on the Model through the Meta settings.
Otherwise the record is saved and (accessible by `Model.objects.all()`) is
emitted. This is based on a dedicated emitter, like the MySQL `INSERT`
statement generator. Emitting provides some of the following features:

#. **trim** if string and max_length is set (note the full string is stored in the intermediate object!).
#. **validate** the output (if output_validate is provided).
#. **replace** the value with some output string (if provided).
#. **anonymize** has been added to the output as of version 0.6.0
#. **write** in a dedicated format as dictated by the emitter.
