Fred Akalin

A Pair of Counterexamples in Vector Calculus

2011-11-27T00:00:00-08:00

While recently reviewing some topics in vector calculus, I became curious as to why violating seemingly innocuous conditions for some theorems leads to surprisingly wild results. In fact, I was struck by how these theorems resemble computer programs, not in some abstract way, but in how the lack of “input validation” leads to non-obvious behavior in the face of erroneous input.

I found that understanding why these counterexamples lead to wild results deepened my understanding of the theorems involved and their proofs. Besides, pathological examples are more interesting than well-behaved ones!

First, let's look at a “counterexample” to Green's theorem:

1. Two functions $L, M \colon \RR^2 \to \RR$ and a positively-oriented, piecewise smooth, simple closed curve $C$ in $\RR^2$ enclosing the region $D$ such that \[ \oint_C L \,dx + M \,dy \ne \iint_D \left( \pd{M}{x} - \pd{L}{y} \right) \,dx \,dy \text{.} \]

Let \begin{align*} L &= -\frac{y}{x^2+y^2} \text{, } M = \frac{x}{x^2+y^2} \text{,} \end{align*} and $C$ be a curve going clockwise around the rectangle $D = [-1, 1]^2$. Then the integral of $L \,dx + M \, dy$ around $C$ is $2 \pi$ since it encloses the origin. But \[ \pd{M}{x} = \pd{L}{y} = \frac{y^2-x^2}{x^2+y^2} \] so the difference of the two vanishes everywhere but the origin, where neither function is defined. Therefore, the (improper) integral over $D$ also vanishes, proving the inequality. ∎

Of course, the easy explanation is that the discontinuity of $L$ and $M$ at the origin violates a condition of Green's theorem. But that doesn't really tell us anything, so let's break down the theorem and see where exactly it fails.

Green's theorem is usually proved first for rectangles $[a, b] \times [c, d]$, which suffices for our purpose. If $C$ is a curve that goes counter-clockwise around such a rectangle $D$, then we can easily show that \[ \oint_C L \,dx = - \iint_D \pd{L}{y} \,dx \,dy \] and \[ \oint_C M \,dy = \iint_D \pd{M}{x} \,dx \,dy \text{,} \] with the sum of these two formulas proving the theorem.

So the first sign of trouble is that the theorem freely interchanges addition and integration. Since the partial derivatives of our functions diverge at the origin, if $D$ contains the origin then the integrals of those partial derivatives over $D$ may not even be defined, even if the integral of their difference is.

But the problem arises even before that. The statements above are proved by showing \[ \oint_C L \,dx = - \int_a^b \left( \int_c^d \pd{L}{y} \,dy \right) \,dx \] and \[ \oint_C M \,dy = \int_c^d \left( \int_a^b \pd{M}{x} \,dx \right) \,dy \text{.} \] both of which hold for our example. But notice that in one case we integrate with respect to $y$ first, and in the other case we integrate with respect to $x$ first. Therefore, we have to interchange the order of integration or convert to a double integral in order to get them to a form where we can add them. And there's the rub: if $D$ contains the origin, switching the order of integration for either integral above switches the sign of the result!

This fully explains the discrepancy; since the result of both integrals above (with the iteration order preserved) is $\pi$, adding them together as-is gives the expected result of $2 \pi$. But if we switch the iteration order of one of the iterated integrals as done in the proof of Green's theorem, then we switch the result of that integral to $-\pi$, which cancels with the result of the other unchanged integral to produce $0$.

So now let's examine this strange behavior of the sign of an integration's result depending on the iteration order. This leads us to our next “counterexample,” this time for Fubini's theorem:

2. A function $f \colon \RR^2 \to \RR$ whose iterated integrals over a rectangle $D = [a, b] \times [c, d] \subset \RR^2$ differ.

Let \[ f(x, y) = \frac{x^2-y^2}{(x^2+y^2)^2} \quad \text{ and } \quad D = [-1, 1]^2\text{.} \] The two iterated integrals of $f$ over $D$ are usually written as \[ \int_{-1}^1 \left( \int_{-1}^1 f(x, y) \,dy \right) \,dx \qquad \text{ and } \qquad \int_{-1}^1 \left( \int_{-1}^1 f(x, y) \,dx \right) \,dy \] but let's define them more carefully to make it easier to justify our calculations.

Let \begin{align*} u_k &= y \mapsto f(k, y) \\ v_l &= x \mapsto f(x, l) \text{.} \end{align*} In other words, given the real constants $k$ and $l$, construct the (possibly partial) real functions $u_k(y)$ and $v_l(x)$ by partially-evaluating $f$ at $x = k$ and $y = l$, respectively.

Then, if we also let \[ U(x) = \int_{-1}^1 u_x(y) \,dy \qquad \text{ and } \qquad V(y) = \int_{-1}^1 v_y(x) \,dx \text{,} \] we can write the iterated integrals as \[ \int_{-1}^1 U(x) \,dx \qquad \text{ and } \qquad \int_{-1}^1 V(y) \,dy \text{.} \]

Computing $U(x)$ for $x \neq 0$, we get \begin{align*} U(x) &= \int_{-1}^1 \pd{}{y} \left( -\frac{y}{x^2+y^2} \right) \,dy \\ &= \left. -\frac{y}{x^2+y^2} \right|_{y=-1}^{y=1} \\ &= -\frac{2}{x^2+1} \text{.} \end{align*}

Attempting to evaluate $U(0)$, we see that \begin{align*} U(0) &= \int_{-1}^1 \frac{0^2-y^2}{(0^2+y^2)^2} \,dy \\ &= - \int_{-1}^1 \frac{dy}{y^2} \end{align*} which diverges. So \[ U(x) = -\frac{2}{x^2+1} \text{ for } x \ne 0 \text{.} \]

By a similar computation, we find that \[ V(y) = \frac{2}{y^2+1} \text{ for } y \ne 0 \text{.} \]

Since $U(x)$ isn't defined at $0$, we have to treat it as an improper integral, although doing so poses no real difficulty: \begin{align*} \int_{-1}^1 U(x)\,dx &= \lim_{a \nearrow 0} \left( \int_{-1}^a -\frac{2}{x^2+1} \,dx \right) + \lim_{a \searrow 0} \left( \int_{a}^1 -\frac{2}{x^2+1} \,dx \right) \\ &= \lim_{a \nearrow 0} \Bigl( \left. -2 \arctan(x) \right|_{-1}^{a} \Bigr) + \lim_{a \searrow 0} \Bigl( \left. -2 \arctan(x) \right|_{a}^{1} \Bigr) \\ &= \left. -2 \arctan(x) \right|_{-1}^{0} + \left. -2 \arctan(x) \right|_{0}^{1} \\ &= \left. -2 \arctan(x) \right|_{-1}^{1} \\ &= -\pi \text{.} \end{align*}

Similarly, \[ \int_{-1}^1 V(y)\,dy = \pi \text{,} \] so the iterated integrals of $f(x, y)$ over $[-1, 1]^2$ differ; in fact, as we claimed above, switching the iteration order switches the sign of the result. ∎

We can repeat the above calculations for an arbitrary rectangle to see that the iterated integrals of $f(x, y)$ differ if $D$ contains the origin either as an interior point or a corner. But there's an easier way to prove that statement and also gain some insight as to why $f(x, y)$ has this strange property.

Note that the key facts in the above calculations were that $U(x) \lt 0$ for any $x \ne 0$ and $V(y) \gt 0$ for any $y \ne 0$. Therefore, integrating $U(x)$ over any interval on the $x$-axis would produce a negative result and integrating $V(x)$ over any interval on the $y$-axis would produce a positive result, leading to the difference in iterated integrals. This holds more generally; for any $m, n \gt 0$: \[ \int_{-n}^n f(x, y) \,dy < 0 \qquad \text{ and } \qquad \int_{-m}^m f(x, y) \,dx > 0 \text{.} \] Therefore, \[ \int_{-m}^m \left( \int_{-n}^n f(x, y) \,dy \right) \,dx < 0 \qquad \text{ and } \qquad \int_{-n}^n \left( \int_{-m}^m f(x, y) \,dx \right) \,dy > 0 \] so the iterated integrals of $f(x, y)$ differ over the rectangles $[-m, m] \times [-n, n]$. Since any rectangle $D$ containing the origin as an interior point must contain some smaller rectangle $E = [-m, m] \times [-n, n]$, the iterated integrals of $f(x, y)$ over $E$ differ and therefore must also differ over $D$.

Furthermore, since $f(x, y)$ is even in both $x$ and $y$, you can carry out a similar argument to the above with intervals of the form $[0, m]$ or $[-m, 0]$ to show that the iterated integrals of $f(x, y)$ must also differ over any rectangle with the origin as a corner.

So the essential property of $f(x, y)$ is that slicing it along the $x$-axis gives a function which has positive area under the curve on any interval symmetric around $0$ or with $0$ as an endpoint, and that slicing it similarly along the $y$-axis gives a function with has negative area. Therefore, on a rectangle symmetric around the origin or with the origin as a corner, one can choose the sign of the iterated integral by choosing which axis to slice first.

The next thing to investigate is how exactly the iterated integrals of $f(x, y)$ over the rectangle $D$ are expressed such that they differ only when $D$ contains the origin, especially considering that the $f(x, y)$ is expressed in quite a simple form. To do that, let's consider the simple case of a rectangle $D = [\delta, 1] \times [\epsilon, 1]$ where we can vary $\delta$ and $\epsilon$ at will.

Let \begin{align*} I_{yx}(\delta, \epsilon) &= \int_{\delta}^1 \left( \int_{\epsilon}^1 f(x, y) \,dy \right) \,dx \\ I_{xy}(\delta, \epsilon) &= \int_{\epsilon}^1 \left( \int_{\delta}^1 f(x, y) \,dx \right) \,dy \text{.} \end{align*} Then, for $\epsilon \neq 0$: \begin{align*} I_{yx}(\delta, \epsilon) &= \int_{\delta}^1 \left( \int_{\epsilon}^1 \frac{y^2-x^2}{(x^2+y^2)^2} \,dy \right) \,dx \\ &= \int_{\delta}^1 \left( \left. -\frac{y}{x^2+y^2} \right|_{y=\epsilon}^{y=1} \right) \,dx \\ &= \int_{\delta}^1 \Biggl( -\frac{1}{1+x^2} - \left( -\frac{\epsilon}{\epsilon^2+x^2} \right) \Biggr) \,dx \\ &= \int_{\delta}^1 \frac{dx/\epsilon}{1+(x/\epsilon)^2} - \int_{\delta}^1 \frac{dx}{1+x^2} \\ &= \arctan\left(\frac{1}{\epsilon}\right) - \arctan\left(\frac{\delta}{\epsilon}\right) - \frac{\pi}{4} + \arctan(\delta) \text{,} \end{align*} and for $\epsilon = 0$: \[ I_{yx}(\delta, 0) = -\frac{\pi}{4} + \arctan(\delta) \text{.} \] Similarly, for $\delta \neq 0$: \begin{align*} I_{xy}(\delta, \epsilon) &= \int_{\epsilon}^1 \left( \int_{\delta}^1 \frac{y^2-x^2}{(x^2+y^2)^2} \,dx \right) \,dy \\ &= \int_{\epsilon}^1 \left( \left. \frac{x}{x^2+y^2} \right|_{x=\delta}^{x=1} \right) \,dy \\ &= \int_{\epsilon}^1 \left( \frac{1}{1+y^2} - \frac{\delta}{\delta^2+x^2} \right) \,dy \\ &= \int_{\epsilon}^1 \frac{dy}{1+y^2} - \int_{\epsilon}^1 \frac{dy/\delta}{1+(y/\delta)^2} \\ &= \frac{\pi}{4} - \arctan(\epsilon) - \arctan\left(\frac{1}{\delta}\right) + \arctan\left(\frac{\epsilon}{\delta}\right) \text{,} \end{align*} and for $\delta = 0$: \[ I_{xy}(0, \epsilon) = \frac{\pi}{4} - \arctan(\epsilon) \text{.} \] Then let $\Delta = I_{xy} - I_{yx}$ be the difference between the two iterated integrals. We can use the identity \[ \arctan(x) + \arctan\left(\frac{1}{x}\right) = \frac{\pi}{2} \sgn{x} \] to simplify $\Delta(\delta, \epsilon)$ if neither $\delta$ nor $\epsilon$ is zero: \begin{align*} \Delta(\delta, \epsilon) &= \bigl( \pi/4 - \arctan(\epsilon) - \arctan(1/\delta) + \arctan(\epsilon/\delta) \bigr) \\ & \quad \mathbin{-} \bigl( \arctan(1/\epsilon) - \arctan(\delta/\epsilon) - \pi/4 + \arctan(\delta) \bigr) \\ &= \pi/2 - \bigl( \arctan(\epsilon) + \arctan(1/\epsilon) \bigr) \\ & \quad \mathbin{-} \bigl( \arctan(\delta) + \arctan(1/\delta) \bigr) \\ & \quad \mathbin{+} \bigl( \arctan(\delta/\epsilon) + \arctan(\epsilon/\delta) \bigr) \\ &= \frac{\pi}{2} \bigl( 1 - \sgn(\epsilon) - \sgn(\delta) + \sgn(\delta/\epsilon) \bigr) \text{.} \end{align*}

Using the properties of $\sgn(x)$, we can simplify this to the final expression: \[ \Delta(\delta, \epsilon) = \frac{\pi}{2} \bigl( 1 - \sgn(\delta) \bigr) \bigl( 1 - \sgn(\epsilon) \bigr) \] which we can prove still holds if either $\delta$ or $\epsilon$ is zero (or both).

So with the simplified expression for $\Delta(\delta, \epsilon)$, it becomes apparent how $\sgn(x)$ is used to control the value of $\Delta(\delta, \epsilon)$; as long as either $\delta$ or $\epsilon$ is positive, $1 - \sgn(x)$ zeroes out the entire expression.

Understanding Evlis Tail Recursion

2011-10-28T00:00:00-07:00

While reading about proper tail recursion in Scheme, I encountered a similar but obscure optimization called evlis tail recursion. In the paper where it was first described, the author claims it "dramatically improve the space performance of many programs," which sounded promising.

However, the few places where its mentioned don't do much more than state its definition and claim its usefulness. Hopefully I can provide a more detailed analysis here.

Consider the straightforward factorial implementation in Scheme:

(define (fact n) (if (<= n 1) 1 (* n (fact (- n 1)))))

It is not tail-recursive, since the recursive call is nested in another procedure call. However, it's almost tail-recursive; the call to * is a tail call, and the recursive call is its last subexpression, so it will be the last subexpression to be evaluated.

The function that takes a list of expressions, evaluates them, and returns the results as a list is traditionally called evlis, hence the name of the optimization.

Recall what happens when a procedure call (represented as a list of subexpressions) is evaluated: each subexpression is evaluated, and the first result (the procedure) is passed the other results as arguments.

Evlis tail recursion can be described as follows: when performing a procedure call and during the evaluation of the last subexpression, the calling environment is discarded as soon as it is not required. The distinction between evlis tail recursion and proper tail recursion is subtle. Proper tail recursion requires only that the calling environment be discarded before the actual procedure call; evlis tail recursion discards the calling environment even sooner, if possible.

An example will help to clarify things. Given fact as defined above, say you evaluate (fact 10) and you're in the procedure call with n = 5. The call stack of a properly tail-recursive interpreter would look like this:

evalExpr
--------
env = { n: 10 } -> <top-level environment>
expr = '(* n (fact (- n 1)))'
proc = <native function: *>
args = [10, <pending evalExpr('(fact (- n 1))', env)>]

evalExpr
--------
env = { n: 9 } -> <top-level environment>
expr = '(* n (fact (- n 1)))'
proc = <native function: *>
args = [9, <pending evalExpr('(fact (- n 1))', env)>]

...

evalExpr
--------
env = { n: 6 } -> <top-level environment>
expr = '(* n (fact (- n 1)))'
proc = <native function: *>
args = [6, <pending evalExpr('(fact (- n 1))', env)>]

evalExpr
--------
env = { n: 5 } -> <top-level environment>
expr = '(if ...)'

whereas the call stack of an evlis tail-recursive interpreter would look like this:

evalExpr
--------
env = { n: 5 } -> <top-level environment>
pendingProcedureCalls = [
  [<native function: *>, 10],
  [<native function: *>, 9],
  ...
  [<native function: *>, 6]
]
expr = (if ...)

In this implementation, the last subexpression of a procedure call is evaluated exactly like a tail expression, but the procedure call and non-last subexpressions are pushed onto a stack. Whenever an expression is reduced to a simple one and the stack is non-empty, a pending procedure call with its other args are popped off, and it is called with the reduced expression as the final argument.

Note that this didn't change the asymptotic behavior of the procedure; it still takes $\Theta(n)$ memory to evaluate. However, only the bare minimum of information is saved: the list of pending functions and their arguments. Other auxiliary variables, and crucially the nested calling environments, aren't preserved, leading to a significant constant-factor reduction in memory.

This raises the question: Are there cases where evlis tail recursion leads to better asymptotic behavior? In fact, yes; consider the following (contrived) implementation of factorial:

(define (fact2 n)
  (define v (make-vector n))
  (* (n (fact2 (- n 1)))))

Before the main body of the function, a vector of size $n$ is defined. This means that the environments in the call stack of a properly tail-recursive interpreter would look like this:

env = { n: 10, v: <vector of size 10> } -> <top-level environment>
env = { n: 9, v: <vector of size 9> } -> <top-level environment>
env = { n: 8, v: <vector of size 8> } -> <top-level environment>
env = { n: 7, v: <vector of size 7> } -> <top-level environment>
...

whereas the an evlis tail-recursive interpreter would keep around only the current environment. Therefore, the properly tail-recursive interpreter would require $\Theta(n^2)$ memory to evaluate (fact2 n) while the evlis tail-recursive interpreter would require only $\Theta(n)$

Studying examples like the one above enabled me to finally understand how evlin tail recursion worked and what sort of savings it gives. However, I have yet to find a practical example where evlis tail recursion gives the same sort of asymptotic gains as described above, and I'd be interested to receive some. But perhaps the "large gains" mentioned in the various papers describing it are only constant-factor reductions in memory.

In any case, another important difference in Scheme between proper tail recursion and evlis tail recursion is that the former is a language feature and the latter is an optimization. That means that it is acceptable and even encouraged to write Scheme programs that take advantage of proper tail recursion, but it would be unwise to rely on evlis tail recursion for the asymptotic performance of your function. Instead, one should treat it just as a nice constant-factor speed gain.

Note that it is easy to make evlis tail recursion "smarter." Since Scheme doesn't specify the order of argument evaluation, an interpreter could evaluate arguments to maximize the gains from evlis tail recursion. As an easy example, if we had switched the arguments to + in fact above, making it non-evlis-tail-recursive, a smart compiler could still treat it as such. A possible rule of thumb would be to pick a non-trivial function call to evaluate last.

To complete the picture, I will outline below the evaluation function for a simple evlis tail-recursive Scheme interpreter in Javascript. All of the sources I've found describe it in terms of compilers, so I think it'll be useful to have a reference implementation for an interpreter.

Let's say we already have a properly tail-recursive interpreter:

function evalExpr(expr, env) {
  // Fake tail calls with a while loop and continue.
  while (true) {
    // Symbols, constants, quoted expressions, and lambdas.
    if (isSimpleExpr(expr)) {
      // The only exit point.
      return evalSimpleExpr(expr, env);
    }
    // (if test conseq alt)
    if (isSpecialForm(expr, 'if')) {
      expr = evalExpr(expr[1], env) ? expr[2] : expr[3];
      continue;
    }
    // (set! var expr)
    if (isSpecialForm(expr, 'set!')) {
      env.set(expr[1], evalExpr(expr[2], env));
      expr = null;
      continue;
    }
    // (define var expr?)
    if (isSpecialForm(expr, 'define')) {
      env.define(expr[1], evalExpr(expr[2], env));
      expr = null;
      continue;
    }
    // (begin expr*)
    if (isSpecialForm(expr, 'begin')) {
      if (expr.length == 1) {
        expr = null;
        continue;
      }
      // Evaluate all but the last subexpression.
      for (var i = 1; i < expr.length - 1; ++i) {
        evalExpr(expr[i], env);
      }
      expr = expr[expr.length - 1];
      continue;
    }
    // (proc expr*)
    var proc = evalExpr(expr.shift(), env);
    var args = expr.map(function(subExpr) { return evalExpr(subExpr, env); });
    // proc.run() returns its body in result.expr and the environment
    // in which to evaluate it (with all its arguments bound) in
    // result.env.
    var result = proc.run(args);
    expr = result.expr;
    // The only time when env is changed.
    env = result.env;
    continue;
  }
}

Then implementing evlis tail recursion requires only a few changes:

function evalExpr(expr, env) {
  // This is a stack of procedures and their non-final arguments that
  // are waiting for their final argument to be evaluated.
  var pendingProcedureCalls = [];
  while (true) {
    if (isSimpleExpr(expr)) {
      expr = evalSimpleExpr(expr, env);
      // Discard calling environment.
      env = null;
      if (pendingProcedureCalls.length == 0) {
        // No pending procedure calls, so we're done (the only exit
        // point).
        return expr;
      }
      var args = pendingProcedureCalls.pop();
      var proc = args.shift();
      args.push(expr);
      var result = proc.run(args);
      expr = result.expr;
      // Change to new environment (the only time when env is
      // changed).
      env = result.env;
      continue;
    }
    ...
    // Everything else remains the same.
    ...
    // (proc expr*)
    var nonFinalSubExprs =
      exprs.slice(0, -1).map(
        function(subExpr) { return evalExpr(subExpr, env); });
    pendingProcecureCalls.push(nonFinalSubExprs);
    // Evaluate the last subexpression as a tail call.
    expr = expr[expr.length - 1];
    continue;
  }
}

An Elementary Way to Calculate the Gaussian Integral

2011-01-06T00:00:00-08:00

While reading Timothy Gowers's blog I stumbled on Scott Carnahan's comment describing an elegant calculation of the Gaussian integral \[ \int_{-\infty}^{\infty} e^{-x^2} \, dx = \sqrt{\pi}\text{.} \] I was so struck by its elementary character that I imagined what it would be like written up, say, as an extra credit exercise in a single-variable calculus class:

Exercise 1. (The Gaussian integral.) Let \[ F(t) = \int_0^t e^{-x^2} \, dx \text{, }\qquad G(t) = \int_0^1 \frac{e^{-t^2 (1+x^2)}}{1+x^2} \, dx \text{,} \] and $H(t) = F(t)^2 + G(t)$.

Calculate $H(0)$.
Calculate and simplify $H'(t)$. What does this imply about $H(t)$?
Use part b to calculate $F(\infty) = \displaystyle\lim_{t \to \infty} F(t)$.
Use part c to calculate \[ \int_{-\infty}^{\infty} e^{-x^2} \, dx\text{.} \]

Although this is simpler than the usual calculation of the Gaussian integral, for which careful reasoning is needed to justify the use of polar coordinates, it seems more like a certificate than an actual proof; you can convince yourself that the calculation is valid, but you gain no insight into the reasoning that led up to it.

Fortunately, David Speyer's comment solves the mystery; $G(t)$ falls out of doing the integration in Cartesian coordinates over a triangular region. Just for kicks, here's how I imagine an exercise based on this method would look like (this time for a multi-variable calculus class):

Exercise 2. (The Gaussian integral in Cartesian coordinates.) Let \[ A(t) = \iint\limits_{\triangle_t} e^{-(x^2+y^2)} \, dx \, dy \] where $\triangle_t$ is the triangle with vertices $(0, 0)$, $(t, 0)$, and $(t, t)$.

Use the substitution $y = sx$ to reduce $A(t)$ to a one-dimensional integral.
Use part a to calculate $A(\infty) = \lim_{t \to \infty} A(t)$.
Use part b to calculate \[ \int_{-\infty}^{\infty} e^{-x^2} \, dx\text{.} \]
Let \[ F(t) = \int_0^t e^{-x^2} \, dx \qquad\text{ and }\qquad G(t) = \int_0^1 \frac{e^{-t^2 (1+x^2)}}{1+x^2} \, dx \text{.} \] Use part a to relate $F(t)$ to $G(t)$.
Use part d to derive a proof of part c using only single-variable calculus.

Finding the Longest Palindromic Substring in Linear Time

2007-11-28T00:00:00-08:00

Another interesting problem I stumbled across on reddit is finding the longest substring of a given string that is a palindrome. I found the explanation on Johan Jeuring's blog somewhat confusing and I had to spend some time poring over the Haskell code (eventually rewriting it in Python) and walking through examples before it "clicked." I haven't found any other explanations of the same approach so hopefully my explanation below will help the next person who is curious about this problem.

Of course, the most naive solution would be to exhaustively examine all $n \choose 2$ substrings of the given $n$-length string, test each one if it's a palindrome, and keep track of the longest one seen so far. This has complexity $O(n^3)$, but we can easily do better by realizing that a palindrome is centered on either a letter (for odd-length palindromes) or a space between letters (for even-length palindromes). Therefore we can examine all $2n + 1$ possible centers and find the longest palindrome for that center, keeping track of the overall longest palindrome. This has complexity $O(n^2)$.

It is not immediately clear that we can do better but if we're told that an $\Theta(n)$ algorithm exists we can infer that the algorithm is most likely structured as an iteration through all possible centers. As an off-the-cuff first attempt, we can adapt the above algorithm by keeping track of the current center and expanding until we find the longest palindrome around that center, in which case we then consider the last letter (or space) of that palindrome as the new center. The algorithm (which isn't correct) looks like this informally:

Set the current center to the first letter.
Loop while the current center is valid:
1. Expand to the left and right simultaneously until we find the largest palindrome around this center.
2. If the current palindrome is bigger than the stored maximum one, store the current one as the maximum one.
3. Set the space following the current palindrome as the current center unless the two letters immediately surrounding it are different, in which case set the last letter of the current palindrome as the current center.
Return the stored maximum palindrome.

This seems to work but it doesn't handle all cases: consider the string "abababa". The first non-trivial palindrome we see is "a|bababa", followed by "aba|baba". Considering the current space as the center doesn't get us anywhere but considering the preceding letter (the second 'a') as the center, we can expand to get "ababa|ba". From this state, considering the current space again doesn't get us anywhere but considering the preceding letter as the center, we can expand to get "abababa|". However, this is incorrect as the longest palindrome is actually the entire string! We can remedy this case by changing the algorithm to try and set the new center to be one before the end of the last palindrome, but it is clear that having a fixed "lookbehind" doesn't solve the general case and anything more than that will probably bump us back up to quadratic time.

The key question is this: given the state from the example above, "ababa|ba", what makes the second 'b' so special that it should be the new center? To use another example, in "abcbabcba|bcba", what makes the second 'c' so special that it should be the new center? Hopefully, the answer to this question will lead to the answer to the more important question: once we stop expanding the palindrome around the current center, how do we pick the next center? To answer the first question, first notice that the current palindromes in the above examples themselves contain smaller non-trivial palindromes: "ababa" contains "aba" and "abcbabcba" contains "abcba" which also contains "bcb". Then, notice that if we expand around the "special" letters, we get a palindrome which shares a right edge with the current palindrome; that is, the longest palindrome around the special letters are proper suffixes of the current palindrome. With a little thought, we can then answer the second question: to pick the next center, take the center of the longest palindromic proper suffix of the current palindrome. Our algorithm then looks like this:

Set the current center to the first letter.
Loop while the current center is valid:
1. Expand to the left and right simultaneously until we find the largest palindrome around this center.
2. If the current palindrome is bigger than the stored maximum one, store the current one as the maximum one.
3. Find the maximal palindromic proper suffix of the current palindrome.
4. Set the center of the suffix from c as the current center and start expanding from the suffix as it is palindromic.
Return the stored maximum palindrome.

However, unless step 2c can be done efficiently, it will cause the algorithm to be superlinear. Doing step 2c efficiently seems impossible since we have to examine the entire current palindrome to find the longest palindromic suffix unless we somehow keep track of extra state as we progress through the input string. Notice that the longest palindromic suffix would by definition also be a palindrome of the input string so it might suffice to keep track of every palindrome that we see as we move through the string and hopefully, by the time we finish expanding around a given center, we would know where all the palindromes with centers lying to the left of the current one are. However, if the longest palindromic suffix has a center to the right of the current center, we would not know about it. But we also have at our disposal the very useful fact that a palindromic proper suffix of a palindrome has a corresponding dual palindromic proper prefix. For example, in one of our examples above, "abcbabcba", notice that "abcba" appears twice: once as a prefix and once as a suffix. Therefore, while we wouldn't know about all the palindromic suffixes of our current palindrome, we would know about either it or its dual.

Another crucial realization is the fact that we don't have to keep track of all the palindromes we've seen. To use the example "abcbabcba" again, we don't really care about "bcb" that much, since it's already contained in the palindrome "abcba". In fact, we only really care about keeping track of the longest palindromes for a given center or equivalently, the length of the longest palindrome for a given center. But this is simply a more general version of our original problem, which is to find the longest palindrome around any center! Thus, if we can keep track of this state efficiently, maybe by taking advantage of the properties of palindromes, we don't have to keep track of the maximal palindrome and can instead figure it out at the very end.

Unfortunately, we seem to be back where we started; the second naive algorithm that we have is simply to loop through all possible centers and for each one find the longest palindrome around that center. But our discussion has led us to a different incremental formulation: given a current center, the longest palindrome around that center, and a list of the lengths of the longest palindromes around the centers to the left of the current center, can we figure out the new center to consider and extend the list of longest palindrome lengths up to that center efficiently? For example, if we have the state:

<"ababa|??", [0, 1, 0, 3, 0, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?]>

where the highlighted letter is the current center, the vertical line is our current position, the question marks represent unread characters or unknown quantities, and the array represents the list of longest palindrome lengths by center, can we get to the state:

<"ababa|??", [0, 1, 0, 3, 0, 5, 0, ?, ?, ?, ?, ?, ?, ?, ?]>

and then to:

<"abababa|", [0, 1, 0, 3, 0, 5, 0, 7, 0, 5, 0, 3, 0, 1, 0]>

efficiently? The crucial thing to notice is that the longest palindrome lengths array (we'll call it simply the lengths array) in the final state is palindromic since the original string is palindromic. In fact, the lengths array obeys a more general property: the longest palindrome d places to the right of the current center (the d-right palindrome) is at least as long as the longest palindrome d places to the left of the current center (the d-left palindrome) if the d-left palindrome is completely contained in the longest palindrome around the current center (the center palindrome), and it is of equal length if the d-left palindrome is not a prefix of the center palindrome or if the center palindrome is a suffix of the entire string. This then implies that we can more or less fill in the values to the right of the current center from the values to the left of the current center. For example, from [0, 1, 0, 3, 0, 5, ?, ?, ?, ?, ?, ?, ?, ?, ?] we can get to [0, 1, 0, 3, 0, 5, 0, ≥3?, 0, ≥1?, 0, ?, ?, ?, ?]. This also implies that the first unknown entry (in this case, ≥3?) should be the new center because it means that the center palindrome is not a suffix of the input string (i.e., we're not done) and that the d-left palindrome is a prefix of the center palindrome.

From these observations we can construct our final algorithm which returns the lengths array, and from which it is easy to find the longest palindromic substring:

Initialize the lengths array to the number of possible centers.
Set the current center to the first center.
Loop while the current center is valid:
1. Expand to the left and right simultaneously until we find the largest palindrome around this center.
2. Fill in the appropriate entry in the longest palindrome lengths array.
3. Iterate through the longest palindrome lengths array backwards and fill in the corresponding values to the right of the entry for the current center until an unknown value (as described above) is encountered.
4. set the new center to the index of this unknown value.
Return the lengths array.

Note that at each step of the algorithm we're either incrementing our current position in the input string or filling in an entry in the lengths array. Since the lengths array has size linear in the size of the input array, the algorithm has worst-case linear running time. Since given the lengths array we can find and return the longest palindromic substring in linear time, a linear-time algorithm to find the longest palindromic substring is the composition of these two operations.

Here is Python code that implements the above algorithm (although it is closer to Johan Jeuring's Haskell implementation than to the above description):

def fastLongestPalindromes(seq):
    """
    Behaves identically to naiveLongestPalindrome (see below), but
    runs in linear time.
    """
    seqLen = len(seq)
    l = []
    i = 0
    palLen = 0
    # Loop invariant: seq[(i - palLen):i] is a palindrome.
    # Loop invariant: len(l) >= 2 * i - palLen. The code path that
    # increments palLen skips the l-filling inner-loop.
    # Loop invariant: len(l) < 2 * i + 1. Any code path that
    # increments i past seqLen - 1 exits the loop early and so skips
    # the l-filling inner loop.
    while i < seqLen:
        # First, see if we can extend the current palindrome.  Note
        # that the center of the palindrome remains fixed.
        if i > palLen and seq[i - palLen - 1] == seq[i]:
            palLen += 2
            i += 1
            continue

        # The current palindrome is as large as it gets, so we append
        # it.
        l.append(palLen)

        # Now to make further progress, we look for a smaller
        # palindrome sharing the right edge with the current
        # palindrome.  If we find one, we can try to expand it and see
        # where that takes us.  At the same time, we can fill the
        # values for l that we neglected during the loop above. We
        # make use of our knowledge of the length of the previous
        # palindrome (palLen) and the fact that the values of l for
        # positions on the right half of the palindrome are closely
        # related to the values of the corresponding positions on the
        # left half of the palindrome.

        # Traverse backwards starting from the second-to-last index up
        # to the edge of the last palindrome.
        s = len(l) - 2
        e = s - palLen
        for j in xrange(s, e, -1):
            # d is the value l[j] must have in order for the
            # palindrome centered there to share the left edge with
            # the last palindrome.  (Drawing it out is helpful to
            # understanding why the - 1 is there.)
            d = j - e - 1

            # We check to see if the palindrome at l[j] shares a left
            # edge with the last palindrome.  If so, the corresponding
            # palindrome on the right half must share the right edge
            # with the last palindrome, and so we have a new value for
            # palLen.
            if l[j] == d: # *
                palLen = d
                # We actually want to go to the beginning of the outer
                # loop, but Python doesn't have loop labels.  Instead,
                # we use an else block corresponding to the inner
                # loop, which gets executed only when the for loop
                # exits normally (i.e., not via break).
                break

            # Otherwise, we just copy the value over to the right
            # side.  We have to bound l[i] because palindromes on the
            # left side could extend past the left edge of the last
            # palindrome, whereas their counterparts won't extend past
            # the right edge.
            l.append(min(d, l[j]))
        else:
            # This code is executed in two cases: when the for loop
            # isn't taken at all (palLen == 0) or the inner loop was
            # unable to find a palindrome sharing the left edge with
            # the last palindrome.  In either case, we're free to
            # consider the palindrome centered at seq[i].
            palLen = 1
            i += 1

    # We know from the loop invariant that len(l) < 2 * seqLen + 1, so
    # we must fill in the remaining values of l.

    # Obviously, the last palindrome we're looking at can't grow any
    # more.
    l.append(palLen)

    # Traverse backwards starting from the second-to-last index up
    # until we get l to size 2 * seqLen + 1. We can deduce from the
    # loop invariants we have enough elements.
    lLen = len(l)
    s = lLen - 2
    e = s - (2 * seqLen + 1 - lLen)
    for i in xrange(s, e, -1):
        # The d here uses the same formula as the d in the inner loop
        # above.  (Computes distance to left edge of the last
        # palindrome.)
        d = i - e - 1
        # We bound l[i] with min for the same reason as in the inner
        # loop above.
        l.append(min(d, l[i]))

    return l

And here is a naive quadratic version for comparison:

def naiveLongestPalindromes(seq):
    """
    Given a sequence seq, returns a list l such that l[2 * i + 1]
    holds the length of the longest palindrome centered at seq[i]
    (which must be odd), l[2 * i] holds the length of the longest
    palindrome centered between seq[i - 1] and seq[i] (which must be
    even), and l[2 * len(seq)] holds the length of the longest
    palindrome centered past the last element of seq (which must be 0,
    as is l[0]).

    The actual palindrome for l[i] is seq[s:(s + l[i])] where s is i
    // 2 - l[i] // 2. (// is integer division.)

    Example:
    naiveLongestPalindrome('ababa') -> [0, 1, 0, 3, 0, 5, 0, 3, 0, 1]
    
    Runs in quadratic time.
    """
    seqLen = len(seq)
    lLen = 2 * seqLen + 1
    l = []

    for i in xrange(lLen):
        # If i is even (i.e., we're on a space), this will produce e
        # == s.  Otherwise, we're on an element and e == s + 1, as a
        # single letter is trivially a palindrome.
        s = i / 2
        e = s + i % 2

        # Loop invariant: seq[s:e] is a palindrome.
        while s > 0 and e < seqLen and seq[s - 1] == seq[e]:
            s -= 1
            e += 1

        l.append(e - s)

    return l

Note that this is not the only efficient solution to this problem; building a suffix tree is linear in the length of the input string and you can use one to solve this problem but as Johan also mentions, that is a much less direct and efficient solution compared to this one.

A Foray into Number Theory with Haskell

2007-07-06T00:00:00-07:00

I encountered an interesting problem on reddit a few days ago which can be paraphrased as follows:

Find a perfect square $s$ such that $1597s + 1$ is also perfect square.

After reading the discussion about implementing a brute-force algorithm to solve the problem and spending a futile half-hour or so trying my hand at find a better way, someone noticed that the problem was an instance of Pell's equation which is known to have an elegant and fast solution; indeed, he posted a one-liner in Mathematica solving the given problem. However, I wanted to try coding up the solution myself as the Mathematica solution, while succinct, isn't very enlightening since the heavy lifting is already done by a built-in function and an arbitrary constant was used for this particular instance of Pell's equation.

Pell's equation is simply the Diophantine equation $x^2 - dy^2 = 1$ for a given $d$; being Diophantine means that all variables involved take on only integer values. (In our original problem, $d$ is 1597 and we are asked for $y^2$.) The solution involves finding the continued fraction expansion of $\sqrt{d}$, finding the first convergent of the expansion that satisfies Pell's equation, and then generating all other solutions from that fundamental solution. We rule out the trivial solution $x = 1$, $y = 0$ which also implies that if $d$ is a perfect square then there is no solution.

A continued fraction is an expression of the form: \[ x = a_0 + \cfrac{1}{a_i + \cfrac{1}{a_2 + \cfrac{1}{a_3 + \cfrac{1}{\ddots\,}}}}\] where all $a_i$ are integers and all but the first one are positive. The standard math notation for continued fractions is quite unwieldy so from now on we'll use $\left \langle a_0; a_1, a_2, \ldots \right \rangle$ instead of the above.

The theory of continued fractions is a rich and beautiful one but for now we'll just state a few facts:

The continued fraction expansion of a number is (mostly) unique.
The continued fraction expansion of a rational number is finite.
The continued fraction expansion of a irrational number is infinite.
A quadratic surd is a number of the form $\frac{a + \sqrt{b}}{c}$ where $a$, $b$, and $c$ are integers. Except maybe for the first term, the continued fraction expansion of a quadratic surd is periodic; that is, it repeats forever after a certain number of terms. This applies in particular to the square root of an integer.
Truncating an infinite continued fraction to get a finite continued fraction gives (in some sense) an optimal rational approximation to the irrational number represented by the infinite continued fraction.

Given a quadratic surd it is fairly easy to manipulate it into the form $a + \frac{1}{q}$ where $q$ is another quadratic surd. This fact can be used to come up with an algorithm to find the continued fraction expansion of a square root. Wikipedia explains it pretty well so I won't go over it, but here is my Haskell implementation:

sqrt_continued_fraction n = [ a_i | (_, _, a_i) <- mdas ]
    where
      mdas = iterate get_next_triplet (m_0, d_0, a_0)

      m_0 = 0
      d_0 = 1
      a_0 = truncate $ sqrt $ fromIntegral n

      get_next_triplet (m_i, d_i, a_i) = (m_j, d_j, a_j)
          where
            m_j = d_i * a_i - m_i
            d_j = (n - m_j * m_j) `div` d_i
            a_j = (a_0 + m_j) `div` d_j

and here are some examples:

Prelude Main> take 20 $ sqrt_continued_fraction 2
[1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2]

Prelude Main> take 20 $ sqrt_continued_fraction 103
[10,6,1,2,1,1,9,1,1,2,1,6,20,6,1,2,1,1,9,1]

Prelude Main> take 20 $ sqrt_continued_fraction 36
[6,*** Exception: divide by zero

(Note that we're assuming that we won't be called with a perfect square. Also, do you notice anything interesting about the periodic portion of the continued fractions, particularly of $\sqrt{103}$?)

For those who are unfamiliar with Haskell, here's a quick list of key facts:

The first line takes a list of triplets and forms a list of all third elements, which is what we're interested in. (The other two elements of the triplet are auxiliary variables used by the algorithm.)
iterate is a function which takes in another function f, an initial variable x, and returns the infinite list [ x, f(x), f(f(x)), f(f(f(x))), ... ].
Note that Haskell uses lazy evaluation and so this function does not take an infinite amount of time to run; all its elements are evaluated (and memoized) only when needed.
The rest of the function is a straightforward representation of the meat of the algorithm described in the above Wikipedia entry.

It may not be clear what $\sqrt{d}$ and its continued fraction expansion has to do with solving Pell's equation. However, notice that if $x$ and $y$ solve Pell's equation then manipulating Pell's equation to get $\sqrt{d}$ on one side reveals that $\frac{x}{y}$ is a good approximation of $\sqrt{n}$. In fact, it is so good that you can prove that $\frac{x}{y}$ must come from truncating the continued fraction expansion of $\sqrt{d}$.

This leads us to the following: if you have an infinite continued fraction $\left \langle a_0; a_1, a_2, \ldots \right \rangle$ you can truncate it into a finite continued fraction $\left \langle a_0; a_1, a_2, \ldots, a_i \right \rangle$ and simplify it into the rational number $\frac{p_i}{q_i}$. The sequence $\frac{p_0}{q_0}, \frac{p_1}{q_1}, \frac{p_2}{q_2}, \ldots$ forms the convergents of $\left \langle a_0; a_1, a_2, \ldots \right \rangle$ and converges to its represented irrational number.

It turns out you can calculate $p_{i+1}$ and $q_{i+1}$ efficiently from $p_i$, $q_i$, $p_{i-1}$, $q_{i-1}$, and $a_{i+1}$ using the fundamental recurrence formulas (which can be proved by induction). Here is my Haskell implementation:

get_convergents (a_0 : a_1 : as) = pqs
    where
      pqs = (p_0, q_0) : (p_1, q_1) :
            zipWith3 get_next_convergent pqs (tail pqs) as

      p_0 = a_0
      q_0 = 1

      p_1 = a_1 * a_0 + 1
      q_1 = a_1

      get_next_convergent (p_i, q_i) (p_j, q_j) a_k = (p_k, q_k)
          where
            p_k = a_k * p_j + p_i
            q_k = a_k * q_j + q_i

and some more examples:

Prelude Main> take 8 $ get_convergents $ sqrt_continued_fraction 2
[(1,1),(3,2),(7,5),(17,12),(41,29),(99,70),(239,169),(577,408)]

Prelude Main> take 8 $ get_convergents $ sqrt_continued_fraction 103
[(10,1),(61,6),(71,7),(203,20),(274,27),(477,47),(4567,450),(5044,497)]

Prelude Main> take 8 $ get_convergents $ sqrt_continued_fraction 1597
[(39,1),(40,1),(1039,26),(1079,27),(2118,53),(3197,80),(27694,693),(113973,2852)]

Prelude Main> let divFrac (x, y) = (fromInteger x) / (fromInteger y)

Prelude Main> take 8 $ map divFrac $ get_convergents $ sqrt_continued_fraction 2
[1.0,1.5,1.4,1.4166666666666667,1.4137931034482758,1.4142857142857144,1.4142011834319526,1.4142156862745099]

Prelude Main> take 8 $ map divFrac $ get_convergents $ sqrt_continued_fraction 103
[10.0,10.166666666666666,10.142857142857142,10.15,10.148148148148149,10.148936170212766,10.148888888888889,10.148893360160965]

Prelude Main> take 8 $ map divFrac $ get_convergents $ sqrt_continued_fraction 1597
[39.0,40.0,39.96153846153846,39.96296296296296,39.9622641509434,39.9625,39.96248196248196,39.9624824684432]

Here are a few more quick facts to help those unfamiliar with Haskell:

The expression a : as forms a new list from the element a and the existing list as (equivalent to cons in Lisp).
zipWith3 is a function that takes in a function f, three lists a, b, and c of the same (possibly infinite) length n, and forms the new list [ f(a[0], b[0], c[0]), f(a[1], b[1], c[1]), ..., f(a[n], b[n], c[n]) ].
Note that the result of zipWith3 is part of the variable pqs which itself appears (twice!) in the arguments to zipWith3. This is a Haskell idiom and reflects the fact that the recurrence formulas define a convergent in terms of its two previous convergents. A simpler example (using the Fibonacci sequence) can be found in the Wikipedia entry for lazy evaluation.
Haskell has built-in data types for integers of arbitrary size which is necessary as the numerators and denominators of the convergents get large quickly. In fact, Haskell has built-in data types for rational numbers (represented as fractions) but it doesn't help us much here.

Since we are guaranteed that some convergent eventually satisfies Pell's equation, we can write a simple function to generate all convergents, test each one to see if it satisfies Pell's equation, and return the first one we see. Here is the Haskell implementation:

get_pell_fundamental_solution n = head $ solutions
    where
      solutions = [ (p, q) | (p, q) <- convergents, p * p - n * q * q == 1 ]

      convergents = get_convergents $ sqrt_continued_fraction n

Note the use of the Haskell's list comprehension syntax, similar to Python, which expresses what I just described in a matter reminiscent of set notation. Here is the full Haskell program designed so its output may be conveniently piped to bc for verification:

module Main where

import System (getArgs)

sqrt_continued_fraction :: (Integral a) => a -> [a]
{- ... the sqrt_continued_fraction function explained above ... -}

get_convergents :: (Integral a) => [a] -> [(a, a)]
{- ... the get_convergents function explained above ... -}

get_pell_fundamental_solution :: (Integral a) => a -> (a, a)
{- ... the get_pell_fundamental_solution function explained above ... -}

main :: IO ()
main = do
  args <- System.getArgs
  let d      = (read $ head $ args :: Integer)
      (p, q) = get_pell_fundamental_solution d in
    putStr $ "d = " ++ (show d) ++ "\n" ++
             "p = " ++ (show p) ++ "\n" ++
             "q = " ++ (show q) ++ "\n" ++
             "p^2 - d * q^2 == 1\n"

and here is it in action:

$ ./solve_pell 1597
d = 1597
p = 519711527755463096224266385375638449943026746249
q = 13004986088790772250309504643908671520836229100
p^2 - d * q^2 == 1

The solution to the original problem is therefore:
5054112910466227478111803017176109047976100000000.

Now that we've found a method to get a solution, the question remains as to whether it's the only one. In fact it is not, but it is the minimal one, and all other solutions (of which there are an infinite number) can be generated from this fundamental one with a simple recurrence relation as described on the Wikipedia article. My program above can be easily extended to generate all solutions instead of just the fundamental one (I'll leave it to the reader as an exercise).

One remaining question is the efficiency of this algorithm. For simplicity, let's neglect the cost of the arbitrary-precision arithmetic involved and assume that the incremental cost of generating each term of the continued fraction expansion and the convergents is constant. Then the main cost is just how many convergents we have to generate before we find one that satisfies Pell's equation. In fact, it turns out that this depends on the length of the period of the continued fraction expansion of $\sqrt{d}$, which has a rough upper bound of $O(\ln(d \sqrt{d}))$. Therefore, the cost of solving Pell's equation (in terms of how many convergents to generate) for a given $n$-digit number is $O(n 2^{n/2})$. This is pretty expensive already, although it's still much better than brute-force search (which is on the order of exponentiating the above expression). Can we do better? Well, sort of; it turns out the length of the answer is of the same order as the expression above, so any algorithm that explicitly outputs a solution necessarily takes that long. However, if you can somehow factor $d$ into $s d'$, where $s$ is a perfect square and $d'$ is squarefree (i.e., not divisible by any perfect square), then you can solve Pell's equation for the smaller number $d'$ and output the solution for $d'$ as the smaller fundamental solution and an expression raised to a certain power involving it. Note that in general this involves factoring $d$, another hard problem, but for which there exists tons of prior work. An interested reader can peruse the papers by Lenstra and Vardi for more details.

As a final note, one of the things I really like about number theory is that investigating such a simple program can lead you down surprising avenues of mathematics and computational theory. In fact, I've had to omit a lot of things I had planned to say to avoid growing this entry to be longer than it already is. Hopefully, this entry helps someone else learn more about this interesting corner of number theory.