Metadata-Version: 2.1
Name: fathom-web
Version: 3.4.1
Summary: Commandline tools for training Fathom rulesets
Home-page: https://mozilla.github.io/fathom/
Author: Erik Rose
Author-email: erik@mozilla.com
License: MPL
Keywords: machine learning,ml,semantic extraction
Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Development Status :: 5 - Production/Stable
Classifier: License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)
Classifier: Programming Language :: Python :: 3
Requires-Dist: click (<8.0,>=7.0)
Requires-Dist: more-itertools (<9.0,>=8.2)
Requires-Dist: numpy (<2.0,>=1.18.1)
Requires-Dist: filelock (>=3.0.12)
Requires-Dist: scikit-learn (>=0.22.2)
Requires-Dist: selenium (>=3.141.0)
Requires-Dist: tensorboardX (<2.0,>=1.6)
Requires-Dist: torch (<2.0,>=1.0)

========================
Fathom Commandline Tools
========================

This is the commandline trainer for `Fathom <https://mozilla.github.io/fathom/>`_, which itself is a supervised-learning system for recognizing parts of web pages. This package also includes other tools for ruleset development, like ``fathom-extract``, ``fathom-pick``, and ``fathom-test``. `See docs for the trainer here <http://mozilla.github.io/fathom/training.html#running-the-trainer>`_.

Version History
===============

3.4.1
  * Add confusion matrices to ``fathom-train`` and ``fathom-test`` readouts.
  * Catch JS syntax errors and other compile-type errors, and report them in ``fathom-train`` and ``fathom-test``.
  * Catch errors due to the absence of prerequisite commands like ``npm``.
  * Catch and nicely report HTTP server errors during autovectorization rather than just spewing tracebacks. Add ``--delay`` option to ``fathom-train`` and ``fathom-test`` to work around them.
  * Don't spit out ``nan`` for precision or F1 when we don't get any samples right.

3.4
  * Make vectorization automatic. This largely obsoletes ``fathom-list`` and ``fathom-serve``. We also remove the need to have 3 terminal tabs open, running ``yarn watch``, ``yarn browser``, and ``fathom-serve``. We remove the error-prone hardlinking of the ruleset into FathomFox, which breaks when git changes to a new branch with a changed ruleset file. We eliminate the possibility of forgetting to revectorize after changing a ruleset or samples. And finally, we pave the way to dramatically simplify our teaching and documentation.

    We tried to hew to the CLI design of the previous version of the trainer to keep things familiar. Basically, where you used to pass in a vector file, now feel free to pass in a directory of samples instead. If you do, you'll also need to pass in your ruleset file and the trainee ID so we can turn the samples into vectors behind the scenes. You can also keep passing in vector files manually if you want more control in some niche situation, like if you're trying to reproduce results from an old branch.

    Aggressive caching is in place to remove every possible impediment to using auto-vectorization. We store hashes of the ruleset and samples so we can tell when revectorizing is necessary. We also cache a built copy of FathomFox (embedded in the Python package) so we don't need to run npm or yarn or hit the network again until you upgrade to a new version of the Fathom CLI tools.
  * Add an ``--exclude`` option to the trainer to help with feature ablation.
  * Fix an issue where the trainer would read vectors as non-UTF-8 on Windows.
  * In the trainer output, make tag excerpts that contain wide Unicode chars fit in their columns.
  * Don't show tag excerpts in ``fathom-test`` by default.
  * Add application/x-javascript and application/font-sfnt to ``fathom-extract``'s list of known MIME types.
  * ``fathom-list``, though no longer needed in most cases, is now always recursive. It has also learned to ignore ``resources`` directories.
  * ``fathom-unzip`` is gone.

3.3
  * Add to the trainer a readout of the average time per candidate tag examined.
  * Replace trainer's per-page metrics, which were increasingly incoherent in Fathom 3, with per-tag ones. Per-page results were most useful back before Fathom could emit confidences. Now, most problems are concerned with per-tag accuracy, and problems that innately concern the page as a whole model it by scoring the ``<html>`` tag. Thus, we swap out the old per-page report for a per-tag one. This is a superset of the per-page report.
  * Add a confidence-threshold customization option to fathom-train.

3.2
  * Add ``fathom-test`` tool for computing test-corpus accuracies.
  * Add ``fathom-extract`` to break down frozen pages into small enough pieces to check into GitHub.
  * Add ``fathom-serve`` to dodge the CORS errors that otherwise happen when loading extracted pages.
  * Add a test harness for the Python code.
  * Add confidence intervals for false positives and false negatives in trainer.
  * Add precision and recall numbers to trainer.
  * Add optional positive-sample weighting in trainer, for trading off between precision and recall.
  * Add experimental support for deeper neural networks in trainer.
  * Add recognition-time speed metrics to trainer.

3.1
  * Add ``fathom-list`` tool.
  * Further optimize trainer: about 17x faster for a 60-sample corpus, with superlinear improvements for larger ones.

3.0
  * Move to Fathom repo.
  * Add ``fathom-unzip`` and ``fathom-pick``.
  * Switch to the Adam optimizer, which is significantly more turn-key, to the point where it doesn't need its learning-rate decay set manually.
  * Tolerate pages for which no candidate nodes were collected.
  * Add 95% CI for per-page training accuracy.
  * Add validation-guided early stopping.
  * Revise per-page accuracy calculation and display.
  * Shuffle training samples before training.
  * Add false-positive and false-negative numbers to per-tag metrics.

3.0a1
  * First release, intended for use with Fathom itself 3.0 or later


