Interfacing to BCMS CTS
=======================

The British Cattle Movement Service is a division of the UK Rural Payments
Agency. They run a database of all British bovines, known as the Cattle
Tracing System. Farmers record all births, deaths, arrivals and departures,
and can access this information for their animals.

As well as a clunky web interface, CTS has an XML-over-HTTP API. This is a
hilarious showcase of enterprise mediocrity: ugly, inconsistent, mixed-case
tag/attribute names in an XML soup that is base64-encoded and then wrapped
in another XML layer. Would you like an AbstractSingletonProxyFactoryBean
with that?

Implementation highlights include multi-second latency from virus-scanning
the base64 payload before parsing it as XML, and (very) asynchronous
responses to the smallest database modifications. The best one can do in a
client interface is to abstract away some of the ugliness.

Each farmer is automatically issued with a CTS Web Services username and
password, which software needs to authenticate to CTWS. The username is
typically three groups of three digits separated by hyphens in the format
nnn-nnn-nnn. It is not the same as the CTS Online web login.

As a single farmer can have access to multiple holdings, the specific CPH
number must also be provided with requests to prevent ambiguity.

Separately, the DEFRA Data Transfer Service expects software to identify
itself with a fixed username and password. These credentials self-evidently
have no useful security properties as they need to ship with programs in
plaintext-equivalent form, but this flavour of idiocy afflicts a number of
UK public-sector APIs.

Acquiring DDTS credentials is left as an exercise for the user. You can
obtain them officially by jumping through some simple hoops to demonstrate
compliant use of the API on a test server, or extract them from other
software, or just man-in-the-middle any CTWS client's https connection to
read them out.


Installation and configuration
------------------------------

A single Python 3.x script 'cts' is provided. Ensure the #! path is correct
for your system before copying it to a directory on your path.

The DDTS (software) username/password, CTWS (farmer) username/password and
the holding number must be configured either in environment variables or in
the config hash at the top of the 'cts' script.

To use the environment, export DDTSAUTH in the format USERNAME:PASSWORD,
CTWSAUTH in the format USERNAME:PASSWORD, and HOLDING in the format
CC/PPP/HHHH or CC/PPP/HHHH-NN if you have multiple sub-locations associated
with your holding. Each setting is only read from the environment if it is
not set in the config hash in the script.


Listing animals
---------------

To list animals on the holding, run

  cts list

This outputs lines for each animal in the space-separated format

  TAG BREED SEX BORN ARRIVED

where

  - TAG is the registered ear tag, e.g. 'UK701453700119';
  - BREED is the breed code, e.g. 'LIMX' or 'WAX';
  - SEX is a single character 'M' or 'F';
  - BORN is the date of birth in the format YYYY-MM-DD;
  - ARRIVED is the arrival date on the holding in the format YYYY-MM-DD.

This format is chosen to make correct shell, awk or sed one-liners simple.
There are no escape characters or line-continuations, no optional or
potentially empty fields, no fields allowing whitespace, and the date
format can be compared lexicographically without parsing.

For example, to list the group who moved on to the holding on 2021-03-01:

  cts list | awk '$5 == "2021-03-01" { print $0 }'

or to get tags and dates of birth for all heifers/cows born before 2019:

  cts list | awk '$3 == "F" && $4 < "2019-01-01" { print $1, $4 }'

or to generate a full CSV file compatible with spreadsheet software:

  cts list | tr ' ' ','


Listing movement history
------------------------

To list the movement history of animals on the holding, run

  cts history [TAG]...

The list of ear tags can be given as command arguments or on stdin, with one
tag per line. The first whitespace-separated word is assumed to be the ear
tag; any other words on each line are ignored.

This outputs lines of the form

  TAG DATE LOCATION TYPE

where

  - TAG is the registered ear tag, e.g. 'UK701453700119';
  - DATE is the movement date in the format YYYY-MM-DD;
  - LOCATION is the movement location, e.g. '35/166/0188';
  - TYPE describes the moment: 'birth', 'death', 'off' or 'on'.

If an animal is not found on the holding, a single line of the form

  TAG not found

is printed instead. Unfortunately, CTWS fails to distinguish tags that do
not exist from valid animals that have moved to another holding, despite
this information being available in the web interface. However, history is
correctly returned for animals that died on the holding.


Reporting movements
-------------------

To report a movement off the holding, run

  cts move off YYYY-MM-DD [TAG]...

where YYYY-MM-DD is the date of the movement. The list of ear tags can be
given as command arguments or on stdin, with one tag per line. The first
whitespace-separated word is assumed to be the ear tag; any other words on
each line are ignored.

Similarly, to report a movement onto the holding, use

  cts move on YYYY-MM-DD [TAG]...

and to report deaths, use

  cts death YYYY-MM-DD [TAG]...

CTS implements these operations asynchronously: clients submit a request,
get a receipt in return, then call back with the receipt number to retrieve
validation results or an error. In practice, results are usually available
immediately, so for ease of use, cts move combines both operations. It will
automatically retry when requests are backlogged but this is rare.

The receipt number, any validation errors, and the numbers of accepted and
rejected movements are reported on stdout.


Checking for queries
--------------------

To check for queried animals or movements, run

  cts queries

This will report any queries with lines in the format

  animal TAG BREED SEX BORN ARRIVED

or

  move TAG TYPE YYYY-MM-DD

for queried animals and movements respectively. The output should normally
be empty, indicating no problems.


Copying
-------

This software was written by Chris Webb <[email protected]> and is
distributed as Free Software under the terms of the MIT license in COPYING.