A Tour of NTL: NTL past, present, and future

#

A Tour of NTL: NTL past, present, and future

### Some History

Work on NTL started around 1990, when I wanted to implement some new algorithms for factoring polynomials over finite fields. I found that none of the available software was adequate for this task, mainly because the code for polynomial arithmetic in the available software was too slow. So I wrote my own. My starting point was Arjen Lenstra's LIP package for long integer arithmetic, which was written in `C`. It soon became clear that using `C++` instead of `C` would be much more productive and less prone to errors, mainly because of `C++`'s constructors and destructors which allow memory management to be automated. Using `C++` has other benefits as well, like function and operator overloading, which makes for more readable code.

One of the basic design principles of LIP was portability. I adopted this principle for NTL as well, for a number of reasons, not the least of which was that my computing environment kept changing whenever I changed jobs. Achieving portability is getting easier as standards, like IEEE floating point, get widely adopted, and as the definition of and implementations of the `C++` language stabilize.

Since 1990, NTL has evolved in many ways, and it now provides a fairly polished and well-rounded programming interface.

When I started working on NTL, there really were not that many good, portable long integer packages around. Besides LIP, there was the BSD Unix MP library. The first version of GMP was released in the early 1990's. At that point in time, LIP seemed like the best starting point. LIP remains a reasonable long integer package, but in recent years, GMP has really become quite good: it seems well supported on many platforms, and is typically much faster than LIP.

NTL was later re-structured NTL so that one can use GMP in place of the 'traditional' LIP. In fact, this is the default in current NTL releases.

For a more complete history of NTL and details about its evolution and implementation, see my paper "Arithmetic Software Libraries" published as a chapter in Computational Cryptography, and also available online here.

### The Future of NTL

As you can well imagine, there is potentially no end to algorithms one could implement. That is why I have to stop somewhere. I think NTL has reached a point where it provides a reasonably well-rounded suite of algorithms for basic problems. I plan to continue supporting NTL, fixing bugs and improving performance.

While I don't have time to add significant new functionality to NTL, there seems to be an ever-growing number of NTL users out there, and I encourage them to make their code available to others. These might be in the form of NTL "add ons", but there is the possibility of integrating new functionality or algorithmic improvements into NTL itself.

### Wish list

These are a few things I wish others could perhaps contribute to NTL. I'd be happy to discuss and assist with any design and integration issues, or any other ideas for improvement. I'd also be happy to discuss ideas for making NTL more open to make it easier for others to contribute.

- Add more thread boosting to various parts of the library
- Add SIMD support for other architectures besides x86 and Aarch64 (and possibly extend some of the SIMD support for Aarch64)
- Add conversions from `long long` and `unsigned long long` to `ZZ` (and perhaps other types), at least for `C++11`. Note that on most Unix-based platforms, these are the same size as `long`, but some platforms define types such as `int64_t` in terms of the `long long` variants, so even on Unix-based platforms this would be useful
- Reduce linear algebra operations over `ZZ_p` (like matrix inversion, system solving, etc) to matrix multiplication (which is much faster)
  - This should be a fairly straightforward and self contained project
- Fast algorithms for general (or more specifice) multi-point evaluation and interpolation over `ZZ_p` (and other coefficient domains)
  - This should be a fairly straightforward and self contained project
- Bivariate polynomial arithmetic:
  - Multiplication can be implemented by reduction to univariate multiplication vie Kronecker substitution (the logic for this is already written for polynomial multiplication over extension fields, and can be easily repurposed)
  - GCD can be reduced can also be implemented by reduction to univariate polynomials (via multi-point evaluation and interpolation)
  - There are a number of programming interface issues that would have to be worked out to maintain consistency with the rest of NTL
- Matrix arithmetic over polynomials:
  - Many operaations can be reduced to operations over the coefficient domain (via multi-point evaluation and interpolation)
  - There are a number of programming interface issues that would have to be worked out to maintain consistency with the rest of NTL
- Improvements to `zz_pX` arithmetic.
  - For small `p`, it is likely faster to use Kronecker-substitution to reduce `zz_pX` multiplication to `ZZ` multiplication. This is especially true if GMP is used for `ZZ` arithmetic.
  - Implementing this should not be too hard, but then one would have to go through all of the `zz_pX` code to make all other operations directly reduce to multiplication in `zz_pX`. This will be a bit tedious, but it shouldn't be too difficult, since one can copy and paste corresponding code from `zz_pEX`, where this that already been done.
- Improvements to some of the `RR` algorithms.
  - In particular, the trig, exp, and log functions are currently woefully inefficient.