There is a EStackoverflow exception type defined in System.Sysutils but it has been deprecated for several Delphi versions. A stack overflow cannot be trapped reliably via try except and recovering from it is practically impossible. What you can do, however, is to increase the stack size used by the program (in the linker part of the program options). Of course that is a band aid that does not protect from future failures due to an even more pathological set of input data.
You can look into rewriting the algorithm to avoid recursion. Intead of relying on local variables to hold intermediate results (i.e. store them on the stack) you store this data into a record and manage a stack of these records yourself. This gives you much more control over the memory used, as well as removing the hard limit imposed by the default stack size.