2 Main results

A monotone triangle is a triangular array of integers of the following form

$$\begin{align*}\begin{array}{ccccccccccc} & & & & & m_{1,1} & & & & & \\ & & & & m_{2,1} & & m_{2,2} & & & & \\ & & & \dots & & \dots & & \dots & & & \\ & & m_{n-2,1} & & \dots & & \dots & & m_{n-2,n-2} & & \\ & m_{n-1,1} & & m_{n-1,2} & & \dots & & \dots & & m_{n-1,n-1} & \\ m_{n,1} & & m_{n,2} & & m_{n,3} & & \dots & & \dots & & m_{n,n} \end{array} \end{align*}$$

with weak increase along $\nearrow $ - and $\searrow $ -diagonals, and strict increase along rows, that is, $m_{i+1,j} \le m_{i,j} \le m_{i+1,j+1}$ and $m_{i,j} < m_{i,j+1}$ . There is the following simple bijection between monotone triangles with bottom row $1,2,\ldots ,n$ and $n \times n$ ASMs: Let $A=(a_{i,j})_{1 \le i,j \le n}$ be an $n \times n$ ASM and consider the matrix obtained by adding to each entry all the entries that are in the same column above, that is, $S=(\sum _{i'=1}^{i} a_{i',j})_{1 \le i, j \le n}$ . It is not hard to see that this always results in a $\{0,1\}$ -matrix with i occurrences of $1$ ’s in the i-th row. The corresponding monotone triangle is obtained by recording row by row the columns of the $1$ ’s in S in n centered rows. To give an example, we apply this to the following matrix, which is obtained from the matrix in the introduction by rotating. (Note that the matrix in the introduction exhibits a vertical symmetry.)

$$\begin{align*}\begin{pmatrix} 0 & 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 1 &0 & -1 & 1 & 0 \\ 0 & 1& -1 & 1 & 0 &0 & 0 \\ 1 &-1 & 1 &-1 & 1 &-1 & 1 \\ 0 & 1 & -1& 1&0 & 0 & 0 \\ 0 & 0 & 1 &0&-1 & 1 & 0 \\ 0 & 0 & 0 & 0 & 1 & 0 & 0 \end{pmatrix} \longleftrightarrow \begin{pmatrix} 0 & 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 1 &0 & 0 & 1 & 0 \\ 0 & 1& 0 & 1 & 0 &1 & 0 \\ 1 &0 & 1 &0 & 1 &0 & 1 \\ 1 & 1 & 0& 1&1 & 0 & 1 \\ 1 & 1 & 1 &1&0 & 1 & 1 \\ 1 & 1 & 1 & 1 & 1 & 1 & 1 \end{pmatrix} \longleftrightarrow \begin{array}{ccccccccccccc} & & & & & & 5 & & & & & & \\ & & & & & 3 & & 6 & & & & & \\ & & & & 2 & & 4 & & 6 & & & & \\ & & & 1 & & 3 & & 5 & & 7 & & & \\ & & 1 & & 2 & & 4 & & 5 & & 7 & & \\ & 1 & & 2 & & 3 & & 4 & & 6 & & 7 & \\ 1 & & 2 & & 3 & & 4 & & 5 & & 6 & & 7 \end{array} \end{align*}$$

An observation that will be useful later is the following: By rotating and considering only the top n rows of the monotone triangles (which take care of the “fundamental domain”), $(2n+1) \times (2n+1)$ VSASMs correspond to monotone triangles with bottom row $2,4,6,\ldots ,2n$ , or, equivalently, when subtracting $2$ from each entry, with bottom row $0,2,\ldots ,2n-2$ . In order to see this, it is crucial to observe that the central column of a VSASM is always $(1,-1,1,-1,\ldots ,-1,1)^T$ , which is not very difficult.

The following decorated versions of monotone triangles have first appeared in [Reference Fischer and Schreier-Aigner10]:

Definition 2.1. An arrowed monotone triangle (AMT) is a monotone triangle where each entry e carries a decoration from $\{\nwarrow , \nearrow , \nwarrow \!\!\!\!\!\;\!\! \nearrow \}$ such that the following two conditions are satisfied:

• • If e has a $\nwarrow $ -neighbor and is equal to it, then e must carry $\nearrow $ .

• • If e has a $\nearrow $ -neighbor and is equal to it, then e must carry $\nwarrow $ .

In summary, an arrow indicates that the entry is different from the next entry in the specified direction.

We assign the following weight to an arrowed monotone triangle with n rows

$$ \begin{align*} u^{\# \nearrow} v^{\# \nwarrow} w^{\# \nwarrow \!\!\!\!\!\;\!\! \nearrow} \prod_{i=1}^n X_i^{\text{(sum of entries in row } i)-\text{(sum of entries in row } i-1)+(\# \nearrow \text{ in row } i) - (\# \nwarrow \text{ in row } i}), \end{align*} $$

where the sum of entries in row $0$ is defined to be $0$ .

In our examples, we write $^{\nwarrow } e, e^{\nearrow }, ^{\nwarrow }e^{\nearrow }$ if entry e is decorated with $\nwarrow , \nearrow , \nwarrow \!\!\!\!\!\;\!\! \nearrow $ , respectively. Such an example is provided next:

$$\begin{align*}\begin{array}{ccccccccccccccccc} & & & & & & & & {}^{\nwarrow}4^{\nearrow} & & & & & & & & \\ & & & & & & & {}^{\nwarrow}4 & & {}^{\nwarrow}5 & & & & & & & \\ & & & & & & {}^{\nwarrow}3^{\nearrow} & & 4^{\nearrow} & & {6}^{\nearrow} & & & & & & \\ & & & & & 2^{\nearrow} & & 3^{\nearrow} & & {}^{\nwarrow}{6} & & {}^{\nwarrow}8^{\nearrow} & & & & & \\ & & & & {}^{\nwarrow}1 & & 2^{\nearrow} & & {}^{\nwarrow}5^{\nearrow} & & 6^{\nearrow} & & {}^{\nwarrow}9 & & & & \\ & & & {}^{\nwarrow}0 & & {}^{\nwarrow}2 & & {}^{\nwarrow}4^{\nearrow} & & {}^{\nwarrow}6 & & {}^{\nwarrow}8^{\nearrow} & & 10^{\nearrow} & & & \end{array} \end{align*}$$

Its weight is

$$ \begin{align*} u^7 v^8 w^6 X_1^{4} X_2^{3} X_3^{6} X_4^{7} X_5^{4} X_6^{5}. \end{align*} $$

In the case $n=2$ , there are three undecorated monotone triangles with bottom row $(0,2)$ , which amount to a total of $45$ such monotone triangles with decorations. We list these arrowed monotone triangles, where $^{*} e^{*}$ means that the entry e can be decorated with any of the arrows $\nwarrow , \nearrow , \nwarrow \!\!\!\!\!\;\!\! \nearrow $ :

$$ \begin{align*} \begin{array}{ccc} & {}^{*}0^{*} & \\ {}^{\nwarrow}0 & & {}^{*}2^{*} \\ \end{array}\qquad\quad\begin{array}{ccc} & {}^{*}1^{*} & \\ {}^{*}0^{*} & & {}^{*}2^{*} \\ \end{array}\qquad\quad\begin{array}{ccc} & {}^{*}2^{*} & \\ {}^{*}0^{*} & & 2^{\nearrow} \\ \end{array} \end{align*} $$

The generating function is

(2.1) $$ \begin{align} v X_2 (u X_1 + v X_1^{-1} + w) (u X_2 + v X_2^{-1} + w) + & X_1 X_2 (u X_1 + v X_1^{-1} + w) (u X_2 + v X_2^{-1} + w) ^2 \nonumber\\ & + u X_1^2 X_2 (u X_1 + v X_1^{-1} + w) (u X_2 + v X_2^{-1} + w). \end{align} $$

The significance of arrowed monotone triangles originally stems from the following specialization: Setting $u=v=1$ , $w=-1$ and $X_i=1$ for $i=1,\ldots ,n$ , the generating function of arrowed monotone triangles with bottom row $k_1,\ldots ,k_n$ is the number of monotone triangles with bottom row $k_1,\ldots ,k_n$ . Indeed, if we want to decorate a given monotone triangle with elements from $\{\nwarrow , \nearrow , \nwarrow \!\!\!\!\!\;\!\! \nearrow \}$ in an eligible way, the arrows are actually prescribed for all entries except for those that are different from their $\nwarrow $ - and $\nearrow $ -neighbors (if they exist). In this case, we are free to choose any decoration. We say that such an entry is free. Thus, suppose f is the number of free entries for a given monotone triangle, then there are $3^f$ associated arrowed monotone triangles. Setting $X_i=1$ for $i=1,\ldots ,n$ in the generating function, the contribution of these $3^f$ arrowed monotone triangles in the generating function is up to a monomial in u and v (coming from the entries that are not free) equal to $(u+v+w)^f$ , which evaluates to $1$ if $u=v=1$ and $w=-1$ .

We now examine how the generating function of AMTs relates to three well-known statistics on VSASMs: the number of $-1$ s, the inversion number and the complementary inversion number. Let $A=(a_{i,j})_{1 \le i,j \le 2n+1}$ be a $(2n+1) \times (2n+1)$ VSASM. We denote by $\mathcal {N}(A)$ the number of $-1$ s in the n leftmost columns of A and define

to be the inversion number and the complementary inversion number, respectively. The inversion number of ordinary ASMs was first introduced in [Reference Robbins and Rumsey24] as a generalization of the inversion number of permutation matrices. We define the weight of A to be

$$ \begin{align*} (u+v+w)^{\mathcal{N}(A)} u^{\operatorname{\mathrm{inv}}(A)} v^{\operatorname{\mathrm{inv}}'(A)}. \end{align*} $$

It follows from an argument in [Reference Fischer and Schreier-Aigner9] that the generating function of arrowed monotone triangles with bottom row $0,2,\ldots ,2n-2$ after setting $X_i=1$ for $i=1,\dots ,n$ equals

$$ \begin{align*} (u+v+w)^n \sum_{A} (u+v+w)^{\mathcal{N}(A)} u^{\operatorname{\mathrm{inv}}(A)} v^{\operatorname{\mathrm{inv}}'(A)}, \end{align*} $$

where we sum over all $(2n+1) \times (2n+1)$ VSASMs.

Before we present our first main result, we need two definitions. A column-strict plane partition (CStrPP) is a filling of a Young diagram with positive integers that weakly decrease along rows and strictly decrease down columns, whereas a row-strict plane partition (RStrPP) is a filling of a Young diagram with positive integers that weakly decrease down columns and strictly along rows. Next we display a CStrPP (left) and an RStrPP (right):

Theorem 2.2. For $n \ge 1$ , the generating function of arrowed monotone triangles with bottom row $0,2,\ldots ,2n-2$ is equal to the generating function of pairs $(P,Q)$ of plane partitions of the same shape with n rows (allowing also rows of length zero) such that P is a CStrPP and Q is an RStrPP, and the entries of P in the i-th row from the bottom are no greater than $2i$ , while the entries of Q in the i-th row from the bottom are no greater than i. The weight of such a pair is given by the following monomial:

$$ \begin{align*} w^{\binom{n+1}{2}-\# \text{ of entries in } P} \prod_{i=1}^{n} X_i^{n-1} (u X_i)^{\# \text{ of } 2i-1 \text{ in } P} (v X_i^{-1})^{\# \text{ of } 2i \text{ in } P}. \end{align*} $$

Letting $n=6$ , the example above is a pair that has the properties described in Theorem 2.2. The weight is

$$ \begin{align*} w^{\binom{7}{2} - 16} X_1^5 X_2^5 X_3^5 X_4^5 X_5^5 X_6^5 (u X_1)^0 (u X_2)^2 (u X_3)^2 & (u X_4)^2 (u X_5)^1 (u X_6)^0 \\ & \times (v X_1^{-1})^2 (v X_2^{-1})^3 (v X_3^{-1})^3 (v X_4^{-1})^1 (v X_5^{-1})^0 (v X_6^{-1})^0, \end{align*} $$

which is equal to $w^{5} u^{7} v^{9} X_1^3 X_2^4 X_3^4 X_4^6 X_5^6 X_6^5$ . Next we list all pairs for the case $n=2$ along with their weights, which indeed add up to (2.1).

It is an open problem to find a bijective proof of this theorem. In fact, there is some similarity with the Cauchy identity, which may give a hint that there is a Robinson–Schensted–Knuth-type correspondence behind this theorem. In the case of the classical Cauchy identity, one side of the identity has a combinatorial interpretation in terms of the generating function of pairs of semistandard tableaux of the same shape, which clearly corresponds to the pairs of plane partitions $(P,Q)$ in Theorem 2.2 in this analogy. The other side of the Cauchy identity is interpreted as the generating function of lexicographically ordered two-line arrays, and they generalize permutations. In Theorem 2.2, this “other” side is the generating function of symmetric ASMs, and ASMs themselves are also generalizations of permutations. Moreover, in view of this relation with the Cauchy identity, another open problem is the following: in Theorem 2.2, only the plane partition P carries a multivariate weight. It is an open problem to find a generalization of Theorem 2.2 that involves a multivariate weight also on Q, possibly on a second set of variables, say, $Y_1,\ldots ,Y_n$ .

Theorem 2.2 is analogous to [Reference Fischer and Schreier-Aigner9, Theorem 1.1], where the latter is about ASMs with no symmetry imposed. Also for that theorem, the corresponding open problems just mentioned are unsolved. Next we phrase Theorem 2.2 in a way that makes the relation to [Reference Fischer and Schreier-Aigner9, Theorem 1.1] more apparent. For this purpose, we show that RStrPPs with at most n rows (allowing also rows of length zero) such that the entries in the i-th row from the bottom are no greater than i are in easy bijective correspondence with $(2n+2) \times (2n+2) \times (2n+2)$ totally symmetric self-complementary plane partitions: After replacing each part p of the RStrPP by $n+1-p$ and conjugating, we obtain a semistandard Young tableau with entries in $\{1,2,\ldots ,n\}$ such that the entries in column i (counted from the left) are no smaller than i. Phrased differently, entries i can only occur in the first i columns. For the example preceding Theorem 2.2, we obtain

when choosing $n=6$ .

Now we use the standard procedure to transform semistandard Young tableaux with entries in $\{1,2,\ldots ,n\}$ into Gelfand–Tsetlin patterns with n rows: To obtain the i-th row of the Gelfand–Tsetlin pattern, consider the shape in the semistandard Young tableaux constituted by the entries less than or equal to i, add $0$ ’s to the corresponding partition so that it is of length i and write it in reverse order. In our running example, we obtain the following Gelfand–Tsetlin pattern:

$$\begin{align*}\begin{array}{ccccccccccccccccc} & & & & & & & & 1 & & & & & & & & \\ & & & & & & & 1 & & 2 & & & & & & & \\ & & & & & &1 & & 2 & & 3 & & & & & & \\ & & & & & 0 & & 2 & & 2 & & 4 & & & & & \\ & & & & 0 & & 1 & & 2 & & 3 & & 5 & & & & \\ & & & 0 & & 0 & & 2 & & 3 & & 5 & & 6 & & & \\ \end{array} \end{align*}$$

As the entries less than or equal to i are confined to the first i columns, it follows that all parts in the partition that constitute the i-th row of the Gelfand–Tsetlin pattern are less than or equal to i. We obtain a Gelfand–Tsetlin pattern with n rows, parts in $\{0,1,\ldots ,n\}$ such that the entries in i-th $\nearrow $ -diagonal are no greater i. Adding $1$ to each entry and prepending a $\nearrow $ -diagonal of $1$ ’s on the left results in a Magog triangle of order $n+1$ as defined in [Reference Zeilberger26, Reference Fischer7]. In our example, we obtain

$$\begin{align*}\begin{array}{ccccccccccccccccc} & & & & & & & 1 & & & & & & & & & \\ & & & & & & 1 & & 2 & & & & & & & & \\ & & & & & 1 & & 2 & & 3 & & & & & & & \\ & & & & 1 & &2 & & 3 & & 4 & & & & & & \\ & & & 1 & & 1 & & 3 & & 3 & & 5 & & & & & \\ & &1 & & 1 & & 2 & & 3 & & 4 & & 6 & & & & \\ &1 & & 1 & & 1 & & 3 & & 4 & & 6 & & 7 & & & \end{array}. \end{align*}$$

Such Magog triangles are known to be in bijective correspondence with totally symmetric self-complementary plane partitions [Reference Mills, Robbins and Rumsey20] inside a $(2n+2) \times (2n+2) \times (2n+2)$ box. For a given totally symmetric self-complementary plane partitions T, let $\lambda (T)$ be the shape of the corresponding RStrPP under the aforementioned bijection.

On the other hand, the CStrPPs with the constraints as they appear in the theorem are in easy bijective correspondence with symplectic tableaux as defined in [Reference Koike and Terada14, Section 4]. The generating functions of symplectic tableaux of shape $\lambda $ are known as the symplectic Schur polynomials $\mathrm {sp}_\lambda $ , which admit the following bialternant formula

$$ \begin{align*} \mathrm{sp}_{\lambda} (X_1,\dots,X_n,X_1^{-1},\dots,X_n^{-1}) = \frac{\det_{1 \le i,j \le n} \left(X_i^{\lambda_j + n - j + 1} - (X_i^{-1})^{\lambda_j + n - j + 1} \right)}{\det_{1 \le i,j \le n} \left(X_i^{n - j + 1} - (X_i^{-1})^{n - j + 1} \right)}. \end{align*} $$

Thus, Theorem 2.2 can also be seen as to provide the expansion of the generating function of arrowed monotone triangles with bottom row $0,2,\ldots ,2n-2$ into symplectic characters when setting $u=v=1$ . This was conjectured by Florian Schreier-Aigner [Reference Schreier-Aigner25] independently.

Corollary 2.3. The generating function of AMTs with bottom row $(0,2,\dots ,2n-2)$ is

$$\begin{align*}\prod_{i=1}^{n} X_i^{n-1} \sum_{T} w^{\binom{n+1}{2}-|\lambda(T)|} \mathrm{sp}_{\lambda(T)}(u X_1,\ldots, u X_n, v X_1^{-1},\ldots,v X_n^{-1}), \end{align*}$$

where we sum over totally symmetric self-complementary plane partitions T inside a $(2n+2) \times (2n+2) \times (2n+2)$ box.

Our other main result concerns nonintersecting lattice paths objects that have the same generating function as arrowed monotone triangles with bottom row $0,2,\ldots ,2n-2$ that can be related combinatorially to the lozenge tilings when specializing. Before we state the results, recall that cyclically and vertically symmetric lozenge tilings of a hexagon with side lengths $2n+2,2n,2n+2,2n,2n+2,2n$ and a central triangular hole of size $2$ are in easy bijective correspondence with families of n nonintersecting lattice paths. An example of such a lozenge tiling is provided in Figure 1 (left). Due to the symmetries of the tilings, the holey hexagon can be decomposed into six equal parts, up to rotation and reflection. Figure 1 (right) shows the family of nonintersecting lattice paths that corresponds to the fundamental area. At the end of Section 4, we provide more details and show how these families of nonintersecting lattice paths relate to the following theorem.

The theorem involves lattice paths in $\mathbb {Z}^2$ , see Figure 2 for an example. We divide the lattice points of $\mathbb {Z}^2$ into even and odd points depending on whether the sum of the coordinates is even or odd, respectively.

Figure 2 Example of the lattice paths in Theorem 2.4 for $n=6$ . The associated permutation is $\sigma = (1\;2\;3\;6\;4\;5)$ and the weight is $(-u v)^5 w^9 X_1^5 X_2^5 X_3^5 X_4^5 X_5^5 X_6^5(u X_3+v X_3^{-1}) (u X_6 + v X_6^{-1})$ . In the second region, we draw the even and odd paths in different colors.

Next we list all families of lattice paths from Theorem 2.4 for the case $n=2$ along with their weights up to the overall factor of $X_1 X_2$ .

It can be checked that the weights indeed add up to the generating function in (2.1).

At the end of Section 4, we show that, when setting $u=v=1$ , $w=-1$ and $X_i=1$ for $i=1,\ldots ,n$ , the number of families of lattice paths as described in Theorem 2.4 are equinumerous with the families of nonintersecting lattice paths that correspond to cyclically and vertically symmetric lozenge tilings of a hexagon with alternating side lengths $2n+2$ and $2n$ and a central triangular hole of size $2$ . We provide a purely combinatorial proof of this result in Section 6.

Remark 2.5. Building on the work of Lalonde [Reference Lalonde17], Krattenthaler [Reference Krattenthaler13] showed that descending plane partitions with entries less than or equal to $2n+1$ can be realized as cyclically symmetric lozenge tilings of a hexagon with alternating side length $2n$ and $2n+2$ and central triangular hole with side length $2$ . Let us recall the definition of a descending plane partition. A descending plane partition is the filling of a shifted Young diagram with positive integers such that

• • entries are weakly decreasing along rows and strictly decreasing along columns and

• • the first entry of each row does not exceed the length of the preceding row (if it exists) but is at least the length of its own row.

Figure 3 (right) depicts an example of a descending plane partition.

Figure 3 An example of the bijective correspondence between cyclically symmetric lozenge tilings of a holey hexagon and descending plane partitions. The lozenge tiling on the left side is the same as in Figure 1 but rotated by $30^\circ $ . The dotted lines mark a third of the lozenge tiling as the fundamental area.

To demonstrate the bijective correspondence between descending plane partitions and lozenge tilings, we take the lozenge tiling from Figure 1, rotate it by $30^\circ $ and draw lines as illustrated in Figure 3. We then read off the heights of the tiles, omitting tiles of height $0$ .

While imposing an additional reflective symmetry on the cyclically symmetric lozenge tilings arises naturally from a geometric perspective, characterizing the corresponding descending plane partitions yields a notion that is more obscure than enlightening. Details are provided in [Reference Mills, Robbins and Rumsey19], see in particular Conjecture 3S. For this reason, we choose to work with lozenge tilings rather than descending plane partitions.

3 The generating function of arrowed monotone triangles

We point out that there is a close analogy between arrowed monotone triangles and their generating function with respect to a fixed bottom row on one side and semistandard Young tableaux and Schur polynomials on the other side: Recall that Gelfand–Tsetlin patterns are defined as monotone triangles with the condition of the strict increase along rows dropped. As demonstrated in the introduction, semistandard Young tableaux are in bijective correspondence with Gelfand–Tsetlin patterns. Schur polynomials are multivariate generating functions of semistandard Young tableaux with respect to the weight

$$ \begin{align*} X_1^{\# \text{ of } 1's} X_2^{\# \text{ of } 2's} \cdots X_n^{\# \text{ of } n's}. \end{align*} $$

However, in terms of Gelfand–Tsetlin patterns, this weight can obviously be expressed as

$$ \begin{align*} \prod_{i=1}^n X_i^{\text{(sum of entries in row } i)-\text{(sum of entries in row } i-1)}, \end{align*} $$

which is analogous to the weight we use for arrowed monotone triangles. This analogy extends also to the following formula for the generating function of arrowed monotone triangles with given bottom row, which can be expressed in terms of (extended) Schur polynomials. We define

Note that for $0 \le k_1 \le k_2 \le \ldots \le k_n$ , $s_{(k_n,\ldots ,k_1)}(X_1,\ldots ,X_n)$ is the Schur polynomial of the partition $(k_n,k_{n-1},\ldots ,k_1)$ . In [Reference Fischer and Schreier-Aigner10, Theorem 2.2], the following theorem has been established, which is one of our main ingredients for the present paper.

Theorem 3.1. The generating function of arrowed monotone triangles with bottom row $k_1 < k_2 < \ldots < k_n$ is

$$ \begin{align*} \prod_{i=1}^{n} (u X_i + v X_i^{-1} + w) \prod_{1 \le i < j \le n}\left( u {\operatorname{E}}_{k_i} + v {\operatorname{E}}_{k_j}^{-1} + w {\operatorname{E}}_{k_i} {\operatorname{E}}_{k_j}^{-1} \right) s_{(k_n,k_{n-1},\ldots,k_1)}(X_1,\ldots,X_n), \end{align*} $$

where ${\operatorname {E}}_x$ denotes the shift operator, defined as .

Using the antisymmetrizer, defined as

we rewrite the expression from Theorem 3.1 and obtain

(3.1) $$ \begin{align} & \prod_{i=1}^{n} (u X_i + v X_i^{-1}+w) \prod_{1 \le i < j \le n}\left( u {\operatorname{E}}_{k_i} + v {\operatorname{E}}_{k_j}^{-1} + w {\operatorname{E}}_{k_i} {\operatorname{E}}_{k_j}^{-1} \right) s_{(k_n,k_{n-1},\ldots,k_1)}(X_1,\ldots,X_n) \nonumber\\ &= \prod_{i=1}^{n} (u X_i + v X_i^{-1}+w) \prod_{1 \le i < j \le n}\left( u {\operatorname{E}}_{k_i} + v {\operatorname{E}}_{k_j}^{-1} + w {\operatorname{E}}_{k_i} {\operatorname{E}}_{k_j}^{-1} \right) \frac{\mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{i=1}^{n} X_i^{k_i+i-1} \right] }{\prod_{1 \le i < j \le n} (X_j - X_i)} \nonumber\\ &= \prod_{i=1}^{n} (u X_i + v X_i^{-1}+w) \frac{\mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{1 \le i < j \le n}\left( u {\operatorname{E}}_{k_i} + v {\operatorname{E}}_{k_j}^{-1} + w {\operatorname{E}}_{k_i} {\operatorname{E}}_{k_j}^{-1} \right) \prod_{i=1}^{n} X_i^{k_i+i-1} \right] }{\prod_{1 \le i < j \le n} (X_j - X_i)} \nonumber\\ &= \prod_{i=1}^{n} (u X_i + v X_i^{-1}+w) \frac{\mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{1 \le i < j \le n}\left( u X_i + v X_j^{-1} + w X_i X_j^{-1} \right) \prod_{i=1}^{n} X_i^{k_i+i-1} \right] }{\prod_{1 \le i < j \le n} (X_j - X_i)} \nonumber\\ &= \frac{\mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{1 \le i \le j \le n}\left( u X_j + v X_i^{-1} + w \right) \prod_{i=1}^{n} X_i^{k_i+n-i} \right] }{\prod_{1 \le i < j \le n} (X_j - X_i)}. \end{align} $$

We set $k_i=2i-2$ for all $1\le i \le n$ to restrict to VSASMs. We also multiply with

(3.2) $$ \begin{align} \prod_{1 \le i \le j \le n} \left( u - v X_i^{-1} X_j^{-1} \right), \end{align} $$

which is a symmetric function and can therefore be put inside the antisymmetrizer. Thus, we obtain

(3.3) $$ \begin{align} & \frac{\mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{1 \le i \le j \le n}\left( u X_j + v X_i^{-1} + w \right) \left(u - v X_i^{-1} X_j^{-1} \right) \prod_{i=1}^{n} X_i^{i+n-2} \right] }{\prod_{1 \le i < j \le n} (X_j - X_i)} \nonumber\\ & \qquad\qquad= \prod_{i=1}^{n} X_i^{n-2} \frac{\mathbf{ASym}_{X_1,\ldots,X_n} \left[ \prod_{1 \le i \le j \le n}\left( u X_j + v X_i^{-1} + w \right) \left(u X_j - v X_i^{-1} \right) \right] }{\prod_{1 \le i < j \le n} (X_j - X_i)}. \end{align} $$

The following lemma allows us to manipulate the previous expression even further. It first appeared in [Reference Aigner, Fischer, Konvalinka, Nadeau and Tewari1, Lemma 3.1] with a computational proof based on algebraic manipulations, while in [Reference Fischer and Schreier-Aigner9, Lemma 4.1] also a combinatorial proof is provided. In this paper, we give a new proof which is shorter than the other two.

Lemma 3.2. Let $n \ge 1$ , and $\mathbf {Y}=(Y_1,\ldots ,Y_n), \mathbf {Z}=(Z_1,\ldots ,Z_n)$ be two sets of indeterminants. Then

(3.4) $$ \begin{align} \det_{1 \le i, j \le n} \left( Y_i^j - Z_i^j \right) = \overline{\mathbf{ASym}} \left[ \prod_{1 \le i \le j \le n} (Y_j-Z_i) \right], \end{align} $$

with

Proof. The proof is by induction on n. The result is obvious for $n=1$ . Let $A_n(\mathbf {Y};\mathbf {Z})$ denote the right-hand side of (3.4), and observe that we have

$$ \begin{align*} A_n(\mathbf{Y};\mathbf{Z}) = \sum_{i=1}^{n} (-1)^{i+1} \left( \prod_{k=1}^n (Y_k-Z_i) \right) A_{n-1}(Y_1,\ldots,\widehat{Y_i},\ldots,Y_n;Z_1,\ldots,\widehat{Z_i},\ldots,Z_n), \end{align*} $$

where $\widehat {Y_i}$ and $\widehat {Z_i}$ means that $Y_i$ and $Z_i$ are omitted, respectively. Using the induction hypothesis, it follows that $A_n(\mathbf {Y};\mathbf {Z})$ is

$$ \begin{align*} (-1)^{n-1} \det_{1 \le i, j \le n} \left( \begin{cases} Y_i^j - Z_i^j, & j < n, \\ \prod_{k=1}^n (Y_k-Z_i), & j=n. \end{cases} \right) \end{align*} $$

For $j \in \{1,2,\ldots ,n-1\}$ , we multiply the j-th column with $(-1)^j e_{n-j}(Y_1,\ldots ,Y_n)$ and add it to the last column, where $e_{n-j}$ denotes the elementary symmetric polynomial of degree $n-j$ . The entry in the i-th row and n-th column is then

$$ \begin{align*} & \sum_{j=1}^{n-1} (-1)^j (Y_i^j - Z_i^j) e_{n-j}(Y_1,\ldots,Y_n) + \prod_{k=1}^n (Y_k-Z_i) \\ & \qquad = \sum_{j=1}^{n-1} (-1)^j (Y_i^j - Z_i^j) e_{n-j}(Y_1,\ldots,Y_n) + \sum_{j=0}^n (-1)^j Z_i^j e_{n-j}(Y_1,\ldots,Y_n) \\ & \qquad = \sum_{j=1}^{n-1} (-1)^j Y_i^j e_{n-j}(Y_1,\ldots,Y_n) + e_n(Y_1,\ldots,Y_n) + (-1)^n Z_i^n \\ &\qquad = \prod_{k=1}^n (Y_k - Y_i) - e_n(Y_1,\ldots,Y_n) - (-1)^n Y_i^n + e_n(Y_1,\ldots,Y_n) + (-1)^n Z_i^n \\ & \qquad\qquad\qquad\qquad\qquad\qquad\qquad = (-1)^n \left( Z_i^n - Y_i^n \right), \end{align*} $$

and this proves the assertion.

The crucial observation is that

$$ \begin{align*} \left( u X_j + v X_i^{-1} + w \right) \left(u X_j - v X_i^{-1} \right) = u^2 X_j^2 + u w X_j - v^2 X_i^{-2} - v w X_i^{-1}, \end{align*} $$

and so the lemma is applicable to (3.3); note that for this it was important to multiply with (3.2): Setting $Y_j = u^2 X_j^2 + u w X_j$ and $Z_i = v^2 X_i^{-2} + v w X_i^{-1}$ in the lemma and applying it to (3.3) yields

$$ \begin{align*} \prod_{i=1}^{n} X_i^{n-2} \frac{ \det_{1 \le i,j \le n} \left( \left(u^2 X_i^2 + u w X_i\right)^j - \left(v^2 X_i^{-2} + v w X_i^{-1}\right)^j \right)} {\prod_{1 \le i < j \le n} (X_j - X_i)}. \end{align*} $$

We divide by (3.2) and eventually obtain the following generating function of arrowed monotone triangles with bottom row $0,2,\ldots ,2n-2$ :

(3.5) $$ \begin{align} \prod_{i=1}^{n} X_i^{n-2} (u-v X_i^{-2})^{-1} \frac{\det_{1 \le i,j \le n} \left( \left(u^2 X_i^2 + u w X_i\right)^j - \left(v^2 X_i^{-2} + v w X_i^{-1}\right)^j \right)}{\prod_{1 \le i < j \le n} (X_j - X_i)\left( u - v X_i^{-1} X_j^{-1} \right)}. \end{align} $$

4 Proof of Theorem 2.4

The following lemma is an important tool to transform bialternant formulas into Jacobi–Trudi-type formulas; the proof can be found in [Reference Fischer and Schreier-Aigner10, Lemma 7.2]. The lemma involves complete homogeneous symmetric functions $h_k$ also of negative degree k, which are defined as follows:

Note that $h_{k}(X_1,\ldots ,X_n)$ vanishes for $-n+1 \le k \le -1$ . For $k \le -n$ , they can be expressed in terms of those with positive degree as

(4.1) $$ \begin{align} h_k(X_1,\ldots,X_n) = (-1)^{n+1} (X_1 \cdots X_n)^{-1} & \sum_{l_1+\cdots+l_n=-k-n,l_i \ge 0} X_1^{-l_1} X_2^{-l_2} \cdots X_n^{-l_n} \nonumber\\ & = (-1)^{n+1} (X_1 \cdots X_n)^{-1} h_{-k-n}(X_1^{-1},\ldots,X_n^{-1}). \end{align} $$

In fact, it is not hard to see that the relation

$$ \begin{align*} h_k(X_1,\ldots,X_n) = (-1)^{n+1} (X_1 \cdots X_n)^{-1} h_{-k-n}(X_1^{-1},\ldots,X_n^{-1}) \end{align*} $$

is true for any integer k.

Lemma 4.1. Let $f_j(Y)$ be formal Laurent series for $1 \le j \le n$ , and define

$$ \begin{align*} f_j[Y_1,\ldots,Y_i]=\sum_{k \in \mathbb{Z}} \langle Y^{k} \rangle f_j(Y) \cdot h_{k-i+1}(Y_1,\ldots,Y_i), \end{align*} $$

where $\langle Y^{k} \rangle f_j(Y)$ denotes the coefficient of $Y^{k}$ in $f_j(Y)$ . Then

$$ \begin{align*} \frac{\det_{1 \le i, j \le n} \left( f_j(Y_i) \right) }{\prod_{1 \le i < j \le n} (Y_j - Y_i)} = \det_{1 \le i, j \le n} \left( f_j[Y_1,\ldots,Y_i] \right). \end{align*} $$

We rewrite (3.5) as follows:

(4.2) $$ \begin{align} \prod_{i=1}^{n} X_i^{n-1} \frac{\det\limits_{1 \le i,j \le n} \left( \frac{\left(u^2 X_i^2 + u w X_i\right)^j - \left(v^2 X_i^{-2} + v w X_i^{-1}\right)^j}{u X_i - v X_i^{-1}} \right)}{\prod\limits_{1 \le i < j \le n} \left( (u X_j + v X_j^{-1}) - (u X_i + v X_i^{-1})\right)}. \end{align} $$

We aim at applying Lemma 4.1 to this bialternant with $Y_i=u X_i + v X_i^{-1}$ . For this purpose, we need to express the entries in the i-th row of the matrix that underlies the determinant in (4.2) as formal Laurent series in $Y_i$ , which is possible since these entries are invariant under $X_i \mapsto u^{-1} v X_i^{-1}$ . We rewrite the entries as follows:

(4.3) $$ \begin{align} \frac{\left(u^2 X_i^2 + u w X_i\right)^j - \left(v^2 X_i^{-2} + v w X_i^{-1}\right)^j}{u X_i - v X_i^{-1}} = \sum_{k=0}^j \binom{j}{k} w^{j-k} \frac{u^{j+k} X_i^{j+k} - v^{j+k} X_i^{-(j+k)}}{u X_i - v X_i^{-1}}. \end{align} $$

The next lemma is useful to transform this further.

Lemma 4.2. For $l \ge 0$ , we have

$$ \begin{align*} \frac{u^{l} X^{l} - v^{l} X^{-l}}{u X - v X^{-1}} = \sum_{r=0}^{(l-1)/2} (-uv)^r \binom{l-r-1}{r} (u X + v X^{-1})^{l-2r-1}. \end{align*} $$

Proof. Using the geometric series expansion, the left-hand side can be written as $\sum _{k=0}^{l-1} u^k v^{-k+l-1} X^{2k-l+1}$ . The right-hand side is equal to

$$ \begin{align*} \sum_{r=0}^{(l-1)/2} \sum_{s \ge 0} (-1)^r \binom{l-r-1}{r} & \binom{l-2r-1}{s} u^{r+s} v^{-r-s+l-1} X^{2r+2s-l+1} \\ & = \sum_{k \le l-1} u^{k} v^{-k+l-1} X^{2k-l+1} \sum_{r=0}^{(l-1)/2} (-1)^r \binom{l-r-1}{r} \binom{l-2r-1}{k-r}. \end{align*} $$

Now, using the Chu–Vandermonde identity in the third step, the inner sum simplifies as follows and the proof is complete:

$$ \begin{align*} \sum_{r=0}^{l-k-1} (-1)^r & \frac{(l-r-1)!}{r! (k-r)! (l-r-k-1)!} = \sum_{r=0}^{l-k-1} (-1)^r \binom{k}{r} \binom{l-r-1}{l-r-1-k} \\ & = \sum_{r=0}^{l-k-1} (-1)^{l-1-k} \binom{k}{r} \binom{-k-1}{l-r-1-k} = (-1)^{l-1-k} \binom{-1}{l-1-k} = \begin{cases}1, & k \le l-1, \\0, & k \ge l. \end{cases} \end{align*} $$

We use Lemma 4.2 to rewrite (4.3) as follows:

$$ \begin{align*} \sum_{k=0}^{j} \sum_{r=0}^{(j+k-1)/2} \binom{j}{k} w^{j-k} (-uv)^r & \binom{j+k-r-1}{r} (u X_i + v X_i^{-1})^{j+k-2r-1} \\ & = \sum_{k,r \ge 0} (-uv)^r w^{2j-k-2r-1} \binom{j}{k-j+2r+1} \binom{k+r}{r} (u X_i + v X_i^{-1})^{k}. \end{align*} $$

Applying Lemma 4.1 to (4.2) with $Y_i=u X_i + v X_i^{-1}$ , we have

$$ \begin{align*} & f_j[u X_1+ v X_1^{-1},\ldots,u X_i+ v X_i^{-1}] \\ & = \sum_{k,r \ge 0} (-uv)^r w^{2j-k-2r-1} \binom{k+r}{r} \binom{j}{2j-k-2r-1} h_{k-i+1}(u X_1+ v X_1^{-1},\ldots,u X_i+ v X_i^{-1}) \\ & = \sum_{\substack{k,r \in \mathbb{Z}\\2j-k-2r-1 \ge 0}} (-u v)^r w^k \binom{2j-k-r-1}{r} \binom{j}{k} h_{2j-k-2r-i}(u X_1+ v X_1^{-1},\ldots,u X_i+ v X_i^{-1}). \end{align*} $$

We set $p=2j-k$ and eliminate k. Then we let $q=p-2r$ and eliminate r. Also exchange the role of i and j, which is possible since we are interested in the determinant. We obtain

(4.4) $$ \begin{align} \sum_{p \ge 1} w^{2i-p} \binom{i}{2i-p} \sum_{q \ge 1, 2 \mid (p-q)} (-u v)^{(p-q)/2} \binom{(p+q)/2-1}{(p-q)/2} h_{q-j}(u X_1+ v X_1^{-1},\ldots,u X_j+ v X_j^{-1}), \end{align} $$

which we denote by $a_{i,j} $ .

Figure 4 Example of a lattice path interpretation of (4.4) with $i=5$ , $j=3$ , $p=8$ and $q=6$ . The steps contribute the factor $-u v w^2 (u X_1+v X_1^{-1})^2 (u X_3 + v X_3^{-1})$ to the weight of the path. In the second region, the path is even.

Our combinatorial interpretations are based on the Lindström–Gessel–Viennot lemma [Reference Lindström18, Reference Gessel and Viennot11, Reference Gessel and Viennot12]; we state a version that is useful for us.

Lemma 4.3. Suppose G is a finite directed acyclic edge-weighted graph with weight function ${\operatorname {w}}$ . For $i \in \{1,2,\ldots ,n\}$ , let $A_i, E_i$ be sets of vertices of G and assume that weights are also assigned to the vertices in these sets, also indicated by ${\operatorname {w}}$ . Further, let ${\mathcal P}(A_i,E_j)$ be the generating function of paths from $A_i$ to $E_j$ , that is,

$$ \begin{align*} {\mathcal P}(A_i,E_j) = \sum_{\substack{(x,y) \in A_i \times E_j\\P \text{ path } x \to y}} {\operatorname{w}}(x) {\operatorname{w}}(y) \prod_{{e \in P}} {\operatorname{w}}(e). \end{align*} $$

Then $\det _{1 \le i,j \le n} \left ( {\mathcal P}(A_i,E_j) \right )$ is the signed generating function of families of n pairwise nonintersecting paths $(P_1,\ldots ,P_n)$ with the property that there exists a permutation $\sigma $ of $\{1,2,\ldots , n\}$ (not necessarily the same for all n-tuples) such that $P_i$ is a directed path from a vertex $x_i$ of $A_i$ to a vertex $y_i$ of $E_{\sigma (i)}$ . The “signed” weight is

$$ \begin{align*} \operatorname{\mathrm{sgn}} \sigma \prod_{i=1}^n {\operatorname{w}}(x_i) {\operatorname{w}}(y_i) \prod_{e \in P_i} {\operatorname{w}}(e). \end{align*} $$

We interpret (4.4) as a certain generating function of lattice paths from $(-i,i)$ to $(j-1,-j+2)$ as follows, see Figure 4:

• • In the region $\{(x,y) \mid x \le 0\}$ , the step set is $\{(1,1),(1,0)\}$ . Assuming that $(0,p)$ is the first lattice point on $x=0$ , there are $p-i$ steps of type $(1,1)$ and $2i-p$ steps of type $(1,0)$ , so that there are $\binom {i}{2i-p}$ such paths. Steps of type $(1,0)$ are equipped with the weight w.

• • In the region $\{(x,y) \mid x \ge 0, y \ge 1\}$ , the step set $\{(1,-1),(0,-2)\}$ , and we aim at reaching the line $y=1$ . The steps of type $(0,-2)$ are equipped weight $-uv$ . Assuming that $(q-1,1)$ is the first lattice point on $y=1$ , there are $q-1$ steps of type $(1,-1)$ and $(p-q)/2$ steps of type $(0,-2)$ , so there are $\binom {(p+q)/2-1}{(p-q)/2}$ such paths. Note that the special form of the step set implies $2 \mid (p-q)$ .

• • In the region $\{(x,y) \mid x \ge 0, y \le 1\}$ , the step set is $\{(-1,0),(0,-1)\}$ . Horizontal steps with distance d from the line $y=2$ have weight $u X_d+v X_d^{-1}$ . The generating function is $h_{q-j}(u X_1+ v X_1^{-1},\ldots ,u X_j+ v X_j^{-1})$ as the number of $(-1,0)$ steps is $q-j$ , while the number of $(0,-1)$ steps is $j-1$ .

• • Paths starting in $(-i,i)$ are equipped with the additional weight $X_i^{n-1}$ .

Applying Lemma 4.3 shows that $\prod _{i=1}^{n} X_i^{n-1} \det _{1 \le i, j \le n}(a_{i,j})$ is the generating function of such lattice paths that start in $(-1,1),(-2,2),\ldots ,(-n,n)$ and end in $(0,1),(1,0),\ldots ,(n-1,-n+2)$ , and that are nonintersecting in the first and in the third region. In the second region, we can distinguish between odd and even paths, depending on whether they only reach odd or even lattice points, respectively. In this region, Lemma 4.3 implies that the family of even paths is nonintersecting as well as the family of odd paths. This concludes the proof of Theorem 2.4.

Our final goal in this section is to relate the nonintersecting lattice paths in Theorem 2.4 to cyclically and vertically symmetric lozenge tilings of a hexagon with side lengths $2n+2, 2n, 2n+2, 2n, 2n+2, 2n$ and with a central triangular hole of size $2$ , which are known to be equinumerous with $(2n+1) \times (2n+1)$ VSASMs .

Due to the symmetries of these tilings, we can decompose the hexagon into six fundamental areas, which can be seen in Figure 1. Figure 5 shows such a fundamental area $\mathcal {H}_n$ for $n=5$ . Thus, it suffices to consider lozenge tilings of $\mathcal {H}_n$ which are in bijective correspondence with families of nonintersecting lattice paths with the step set $\{(-1,0),(0,1)\}$ starting in $A_i=(2i-1,i-1)$ and ending in $E_i=(i-1,2i-2)$ for $1 \leq i \leq n$ . An example of this correspondence is illustrated in Figure 1.

Figure 5 Fundamental area $\mathcal {H}_n$ .

The number of families of nonintersecting lattice paths between the two sets $(A_i)_{1 \leq i \leq n}$ and $(E_i)_{1 \leq i \leq n}$ can be calculated by means of the Lindström–Gessel–Viennot lemma (Lemma 4.3):

(4.5) $$ \begin{align} \det_{1 \le i,j \le n} \mathcal{P} (A_i \to E_j) = \det_{1 \le i,j \le n} \left( \binom{i+j-1}{2j-i-1} \right). \end{align} $$

Noting that

(4.6) $$ \begin{align} \left. h_{k}(X_1,\ldots,X_n) \right|{}_{(X_1,\ldots,X_n)=X} = X^k \binom{n+k-1}{k}, \end{align} $$

we observe that, when setting $u=v=1, w=-1$ and $X_i=1$ , for $i=1,2,\ldots ,n$ , then (4.4) simplifies to

(4.7) $$ \begin{align} \sum_{p \ge 1} (-1)^{2i-p} \binom{i}{2i-p} \sum_{q \ge 1,2 \mid (p-q)} 2^{q-j} (-1)^{(p-q)/2} \binom{(p+q)/2-1}{(p-q)/2} \binom{q-1}{j-1}. \end{align} $$

We see that

(4.8) $$ \begin{align} \sum_{p \ge 1} (-1)^{2i-p} \binom{i}{2i-p} \sum_{q \ge 1,2 \mid (p-q)} 2^{q-j} (-1)^{(p-q)/2} \binom{(p+q)/2-1}{(p-q)/2} \binom{q-1}{j-1} = \binom{i+j-1}{2j-i+1}, \end{align} $$

which equals the entries of the matrix underlying the determinant in (4.5).

In Section 6, we provide a combinatorial proof of (4.8) in two steps within our model of lattice paths.

5 Proof of Theorem 2.2

In this section, we see that the signed families of lattice paths from Theorem 2.4 can be transformed, in a purely combinatorial matter, into a signless interpretation in terms of certain plane partitions. Thus, Theorem 2.4 implies Theorem 2.2. In Appendix A, we present a second proof exploiting another Jacobi–Trudi-type expression for (3.5) which differs from the one used in Section 4.

A drawback of Theorem 2.4 is that the lattice path interpretation also involves signs although the generating function has only non-negative coefficients, since it is also the generating function of arrowed monotone triangles. We will define a sign-reversing involution on the families of lattice paths from Theorem 2.4 to see directly that also their generating function has no negative coefficients. In fact, we will arrive again at the signless interpretation in Theorem 2.2. To be more precise, we will first have to “move back” to possibly intersecting paths in the second and third region, and apply the sign-reversing involution on such paths. This sign-reversing involution will also involve a Gessel–Viennot-type sign-reversing involution in the third region.

For this purpose, we will first show that (4.4) has only non-negative coefficients. In fact, we define a sign-reversing involution that shows that the inner sum of (4.4) has no negative coefficients.

Recall that we consider lattice paths from $(0,p)$ to $(j-1,-j+2)$ , with step set $\{(1,-1), (0,-2)\}$ until we reach the line $y=1$ at $(q-1,1)$ such that $q \le p$ , while on and below the line $y=1$ the step set is $\{(-1,0),(0,-1)\}$ . We then equip steps of type $(0,-2)$ with the weight $-uv$ , while steps of type $(-1,0)$ are colored in blue or red, and blue and red steps are equipped with the weight $u X_d$ and $v X_d^{-1}$ , respectively, where d is the distance from the line $y=2$ . All other steps have weight $1$ .

Lemma 5.1. Let $p,j$ be positive integers, then

$$ \begin{align*} \sum_{q \ge 1, 2 \mid (p-q)} (-u v)^{(p-q)/2} \binom{(p+q)/2-1}{(p-q)/2} h_{q-j}(u X_1+ v X_1^{-1},\ldots,u X_j+ v X_j^{-1}) \end{align*} $$

is the generating function of lattice paths from $(0,p)$ to $(j-1,-j+2)$ as given above, but without steps of type $(0,-2)$ and without consecutive pairs of horizontal steps with the first step being blue and the second step being red.

Clearly, forbidding steps of type $(0,-2)$ implies that above the line $y=1$ we go straight from $(0,p)$ to $(p-1,1)$ , using only steps of type $(1,-1)$ .

Using Lemma 5.1, we are left with families of paths with step sets as described in Theorem 2.4 that

• • are nonintersecting in the region $\{(x,y) \mid x \le 0\}$ ,

• • have only diagonal steps $(1,-1)$ in the region $\{(x,y) \mid x \ge 0, y \ge 1\}$ (which implies automatically that the paths are nonintersecting in this region), and

• • consecutive horizontals steps in the region $\{(x,y) \mid x \ge 0, y \le 1\}$ are divided into an initial section of red steps (which may be empty) followed by an ending section of blue steps (which may be empty as well). The paths may still be intersecting in this region at this point.

The lattice point of a maximal section of consecutive horizontal steps that divides the section into two portions of red and blue steps is said to be the center of the section. Note that the center can also be the left or right end point of these maximal sections if there are no blue or no red steps, respectively. Now we say that two paths in the region $\{(x,y) \mid x \ge 0 , y \le 1\}$ touch if they intersect, but their intersections are fully contained in maximal sections of consecutive horizontal steps and none of these intersections contain centers of either paths. Two paths that are intersecting but not touching are said to intersect strongly.

We make the following important observation: Suppose $P_1,P_2$ are two touching paths, and let $(-i_1,i_1),(-i_2,i_2)$ be their starting points, respectively, and $(j_1-1,-j_1+2),(j_2-1,-j_2+2)$ their end points. Then $i_1 \le i_2$ if and only if $j_1 \le j_2$ . This implies that for families where no two paths intersect strongly, the path starting in $(-i,i)$ ends in $(i-1,-i+2)$ , so that the underlying permutation is the identity, which does not contribute to the sign.

Now we define a sign-reversing involution on such families of paths with at least one pair of paths that is intersecting strongly. Among those pairs we chose a canonical one as follows: The starting point of one path $P_1$ should be as rightmost as possible, and among all paths that are intersecting strongly with this path, choose again the path $P_2$ whose starting point is as rightmost as possible.

We distinguish several cases: If $P_1 \cap P_2$ contains a vertical step, then we select the last such vertical step in $P_1$ and interchange the end portions of the two paths after this step. Thus we can assume that $P_1 \cap P_2$ contains only horizontal steps. Then the involution is provided by Figure 6, which is applied to the last connected component of the intersection. In all cases, the sign of the family of paths changes by a factor of $-1$ . An example of families of paths that do not cancel under this involution is given in Figure 7.

Figure 6 The different cases of the sign-reversing involution on strongly intersecting paths.

Figure 7 This family of lattice paths corresponds to the pair of CStrPP and RStrPP in Section 2.

Such paths correspond to pairs $(P,Q')$ of CStrPPs of the same shape with at most n rows with the following properties:

• • The entries of P in the i-th row from the bottom are no greater than $2i$ .

• • The entries of $Q'$ in the i-th row from the bottom are no greater than $2i-1$ and no less than $\ell + i -1$ if $\ell $ is the length of the i-th row.

The entries of P correspond to the heights of the horizontal steps of the paths on and below the line $y=1$ . Each odd entry $2i-1$ corresponds to a red step at distance i from the line $y=2$ and contributes $u X_i$ multiplicatively to the weight, while each even entry $2i$ corresponds to a blue step at distance i from the line $y=2$ and contributes $v X_i^{-1}$ . The entries of $Q'$ correspond to the diagonal steps, more precisely their distance from the line $y=x+1$ , and each of them contributes w to the weight.

Now we subtract $i-1$ from all entries in the i-th row from the bottom of $Q'$ and then $j-1$ from all entries in the j-th column of $Q'$ . We obtain an RStrPP Q with positive entries such that the entries in the i-th row from the bottom are no greater than i. This establishes the proof of Theorem 2.2.

6 Bijective proof of Equation (4.8)

In this section, we present a combinatorial proof of (4.8):

$$ \begin{align*} \sum_{p \ge 1} (-1)^{2i-p} \binom{i}{2i-p} \sum_{q \ge 1,2 \mid (p-q)} 2^{q-j} (-1)^{(p-q)/2} \binom{(p+q)/2-1}{(p-q)/2} \binom{q-1}{j-1} = \binom{i+j-1}{2j-i+1}. \end{align*} $$

In particular, we provide combinatorial proofs within our model of lattice paths for the following two identities:

(6.1) $$ \begin{align} \sum_{q \ge 1, 2 \mid (p-q)} 2^{q-j} (-1)^{(p-q)/2} \binom{(p+q)/2-1}{(p-q)/2} \binom{q-1}{j-1} = \binom{p+j-1}{p-j} \end{align} $$

and

(6.2) $$ \begin{align} \sum_{p \ge 1} (-1)^{2i-p}\binom{i}{2i-p} \binom{p+j-1}{p-j} = \binom{i+j-1}{2j-i+1}. \end{align} $$

Interestingly, we have to leave the regime of nonintersecting lattice paths for the proof of (6.2) and have to work with lattice paths that are possibly intersecting instead. This was not the case in [Reference Fischer and Schreier-Aigner10] and it is an open problem whether we can avoid intersecting lattice paths here as well.

6.1 Combinatorial proof of (6.1)

Recall that in our combinatorial model in Section 4, the left-hand side of (6.1) is interpreted as

• • lattice paths from $(0,p)$ to $(j-1,-j+2)$ ,

• • with step set $\{(1,-1),(0,-2)\}$ until we reach the line $y=1$ (at $(q-1,1)$ ),

• • while on and below the line $y=1$ , the step set is $\{(-1,0),(0,-1)\}$ .

Steps of type $(0,-2)$ are equipped with the weight $-1$ , while steps of type $(-1,0)$ are equipped with the weight $2$ . We take the latter weight into account by coloring such steps either in blue or in red. All steps different from these steps are colored in green to distinguish them from the uncolored paths that will provide an interpretation for the right-hand side of (6.1). The set of these colored paths is denoted by $A(j,p)$ , and we refer to them as the A-paths.

The combinatorial model for the right-hand side of (6.1) are paths from $(0,p)$ to $(2j-1,-j+1)$ with step set $\{(1,-1),(0,-1)\}$ . The set of these uncolored paths is denoted by $B(j,p)$ , and we refer to them as the B-paths.

From $\boldsymbol {B(j,p)}$ to $\boldsymbol {A(j,p)}$ : We explain how to transform paths from $B(j,p)$ into paths from $A(j,p)$ . It is an inductive process. In each step, we will have a lattice path that starts in $(0,p)$ . The initial section will be colored in green (using the step set $\{(1,-1),(0,-2)\}$ of the first region of A-paths), while the ending section will be colored in green, red, and blue (using the step set $\{(0,-1),(-1,0)\}$ of the second region of A-paths). For the middle section, we use the uncolored step set $\{(1,-1),(0,-1)\}$ of B-paths.

When traveling along the path starting at $(0,p)$ , we look for the first pair of uncolored steps $(s_1,s_2)$ and perform the following transformations, see also Figure 8:

1. 1. If $(s_1,s_2)=((1,-1),(1,-1))$ , then we color $s_1$ green, contract $s_2$ , and add a green $(0,-1)$ -step at the end.

2. 2. If $(s_1,s_2)=((1,-1),(0,-1))$ , then we color $s_1$ green, replace $s_2$ by an uncolored $(1,-1)$ -step, and add a blue $(-1,0)$ -step at the end.

3. 3. If $(s_1,s_2)=((0,-1),(1,-1))$ , then we replace $s_1$ by a green $(1,-1)$ -step and add a red $(-1,0)$ -step at the end.

4. 4. If $(s_1,s_2)=((0,-1),(0,-1))$ we make three copies, two of them with the same sign and a third with the opposite sign. In the first two, we replace $s_1$ by a green $(1,-1)$ -step. In both cases, we add a $(-1,0)$ -step at the end; in one case it is blue and in the other case it is red. In the copy with the opposite sign, we replace $(s_1,s_2)$ by a $(0,-2)$ -step. The sign is taken into account, since $(0,-2)$ -steps have sign $-1$ .

Figure 8 Rules for transforming an uncolored B-path into a colored A-path.

If there is only one uncolored step left, delete it. This step is always a step of type $(1,-1)$ as can be seen as follows: Paths in $B(j,p)$ have $2j-1$ steps of this type, and only Rule (1) transforms such steps. Among these $2j-1$ steps, there are $j-1$ pairs of such steps that are transformed using Rule (1), so that there is one such step left in the end. This also shows that we are always in the situation that there is one uncolored step left in the end.

The procedure is well-defined: From the rules, it can also be seen that uncolored $(0,-1)$ -steps are either transformed into green $(1,-1)$ -steps or pairs of them are transformed into green $(0,-2)$ -steps. Suppose that there are $2t$ of the second type so that there are $p-j-2t$ of the first type. It follows that there are $p-2t-1$ green steps of type $(1,-1)$ and t green steps of type $(0,-2)$ . With these steps we reach indeed the line $y=1$ , since

$$\begin{align*}(0,p) + (p-2t-1) \cdot (1,-1) + t \cdot (0,-2) = (p-2t-1,1). \end{align*}$$

Note that we set t such that $p-2t$ corresponds to q in (6.1).

Since none of the rules changes the height of the endpoint of our path except for the deletion of the final uncolored $(1,-1)$ -step, the end point of our path lies on the line $y=-j+2$ . We create (blue or red) steps of type $(-1,0)$ precisely when uncolored $(0,-1)$ steps are transformed, which happens in Rules (2), (3), and (4) using either of the first two options. There are $p-j-2t$ such applications so that the end point lies on the line $x=p-2t-1-(p-j-2t)=j-1$ . Therefore, the end point is $(j-1,-j+2)$ . An example is given in Figure 9.

Figure 9 The transformation of a B-path into a family of A-paths for $p=5$ and $j=2$ .

From $\boldsymbol {A(j,p)}$ to $\boldsymbol {B(j,p)}$ : Conversely, paths from $A(j,p)$ are transformed into paths from $B(j,p)$ as follows: After the first lattice point on the line $y=1$ when coming from $(0,p)$ , we add an uncolored step of type $(1,-1)$ , so that the new end point is $(j,-j+1)$ . From now on, it is again an inductive procedure. In each step, we have a path starting in $(0,p)$ . The starting section uses the step set of $A(j,p)$ for above the line $y=1$ , while the ending section uses the step set of $A(j,p)$ for below the line $y=1$ . The middle part of the path uses the step set of $B(j,p)$ . The rules are as follows (note that they reverse the procedure of how to transform a B-path into an A-path):

1. 1. If the last step of the beginning section is a $(0,-2)$ -step, we replace it by two uncolored $(0,-1)$ -steps.

2. 2. If the last step of the beginning section is a $(1,-1)$ -step and the last step of the path is a green $(0,-1)$ -step, we delete this last step and replace the $(1,-1)$ -step by two uncolored $(1,-1)$ -steps.

3. 3. If the last step of the beginning section is a $(1,-1)$ -step and the last step of the path is a $(-1,0)$ step, we do the following:

   1. (a) If the $(1,-1)$ -step is followed by an uncolored $(0,-1)$ step, we replace the $(1,-1)$ -step by an uncolored $(0,-1)$ step and delete the last step of the path.

   2. (b) If the $(1,-1)$ -step is followed by an uncolored $(1,-1)$ -step, we distinguish according to the color of the last step (which is by assumption a $(-1,0)$ step):

      • (i) If the step is red, we replace the green $(1,-1)$ -step by an uncolored step of type $(0,-1)$ , and delete also the red step.

      • (ii) If the step is blue, we replace the uncolored $(1,-1)$ -step by an uncolored $(0,-1)$ -step, uncolor the green $(1,-1)$ -step, and delete the blue step.

An example is provided in Figure 10.

Figure 10 The transformation of an A-path into a B-path for $p=7$ and $j=2$ .

The procedure is well-defined: It remains to show the following:

• • If the last step of the beginning section is of type $(1,-1)$ then the ending section is not empty.

• • The end point of the final path is $(2j-1,-j+1)$ .

To show the first assertion, note that in each step the number of green $(1,-1)$ -steps is just the number of blue or red $(-1,0)$ -step plus the number of green $(0,-1)$ -steps. This is obviously true for paths in $A(j,p)$ and remains to be true in every step by the nature of the rules: Whenever a green $(1,-1)$ -step is transformed, we delete precisely one step of the other type.

Concerning the change of the end point, note that the end point does not change, except when we are in case (2), in which case the end point is pushed horizontally by one unit to the right. We have such a case (2) for each green $(0,-1)$ -step in the original path in $A(j,p)$ . There are $j-1$ such steps, so that the end point is $(j,-j+1)+(j-1) \cdot (1,0)=(2j-1,-j+1)$ as required.

For this purpose, we will first show that (4.4) has only non-negative coefficients. In fact, our combinatorial proof of (6.1) implies a sign-reversing involution that shows that the inner sum of (4.4) has no negative coefficients. To see this, we introduce essentially the weights from Theorem 2.4 in the combinatorial interpretation of the left-hand side of (6.1) to obtain a combinatorial interpretation of the inner sum of (4.4)

Remark 6.1. The combinatorial proof of (6.1) implies a sign-reversing involution that shows that the inner sum of (6.1) has no negative coefficients. The proof of Lemma 5.1 provides this involution with more general weights. In fact, two A-paths that are paired off under this involution are mapped to the same B-path, but have opposite signs as A-paths.

6.2 Combinatorial proof of (6.2)

Recall that we interpret the left-hand side of (6.2) as paths from $(2j-1,-j+1)$ to $(-i,i)$ . For $x \ge 0$ , we allow steps of type $(-1,1)$ and $(0,1)$ , and, for $x < 0$ , we allow steps of type $(-1,-1)$ and $(-1,0)$ . Steps of type $(-1,0)$ are equipped with the weight $-1$ . Such a path must contain precisely one lattice point, say $(0,p)$ , on the line $x=0$ . We reflect the portion of the path that is to the left of the y-axis along this axis.

This results in paths from $(2j-1,-j+1)$ to $(i,i)$ that touch the line $x=0$ precisely once, namely in $(0,p)$ . From $(2j-1,-j+1)$ to $(0,p)$ , we allow steps of type $(-1,1)$ and $(0,1)$ , and from $(0,p)$ to $(i,i)$ , we allow steps of type $(1,-1)$ and $(1,0)$ . Every step of type $(1,0)$ is equipped with the weight $-1$ . An example is provided in Figure 11.

Next, we construct a sign-reversing involution on a subset of these paths. For this purpose, observe that the point $(0,p)$ divides the path into two subpaths: an upper path that ends in $(i,i)$ and a lower path that starts in $(2j-1,-j+1)$ . Seen as lattice paths, these two paths might coincide in some diagonal steps (in the initial section of the upper path and the ending section of the lower path): Let $k\in \{0,\dots ,i\}$ be the largest integer such that the vertices $(0,p)$ , $(1,p-1)$ , …, $(k,p-k)$ lie both on the upper and the lower path. The point $P=(k,p-k)$ is then the rightmost intersection point of these two paths. If $k < i$ , we perform a transformation on the paths which we depict as follows:

Depending on whether the upper path has a diagonal or a horizontal step after it has reached P, we delete the vertical step of the lower path incident to P and replace the diagonal step of the upper path by the horizontal step or insert a vertical step to the lower path and replace the horizontal step of the upper path by a diagonal step, respectively. This transformation changes the height of the intersection point P and therefore alters the point where the path touches the line $x=0$ . The upper path either gains or loses a horizontal step which changes the weight of the path. Figure 12 illustrates one instance of this sign-reversing involution for $i=4$ and $j=3$ .

Figure 11 Example of the lattice path interpretation of the left-hand side of (6.2) for $i=5$ , $j=3$ and $p=6$ . The weight of the path is $(-1)^4=1$ .

Figure 12 Sign-reversing involution for $i=4$ and $j=3$ .

We leave the path unchanged only if the rightmost intersection point P is $(i,i)$ . In this case, the path touches the line $x=0$ in the point $(0,2i)$ and the upper path as well as the lower path coincide in the i diagonal steps from $(0,2i)$ to $(i,i)$ . We delete those diagonal steps, which leaves us paths from $(2j-1,-j+1)$ to $(i,i)$ with steps of type $(-1,1)$ and $(0,1)$ . These paths are enumerated by $\binom {i+j-1}{2j-i+1}$ .

7 Further types of families of lattice paths

Theorem 2.4 presents a family of lattice paths with the same generating function as arrowed monotone triangles with bottom row $0,2,\ldots ,2n-2$ . In this section, we provide two other families of lattice paths with the same property, the second of which comprises even nonintersecting lattice paths.

7.1 The second type of families of lattice paths

Theorem 7.1. For $n \ge 1$ , the generating function of arrowed monotone triangles with bottom row $0,2,\ldots ,2n-2$ is equal to the signed generating function of n lattice paths with starting points $(1,1),(2,2),\ldots ,(n,n)$ and end points $(2,1),(2,0),\ldots ,(2,-n+2)$ as follows:

• • In the region $\{(x,y) \mid y \ge 1\}$ , the step set is $\{(1,0),(0,-1)\}$ and horizontal steps with distance d from the x-axis are equipped with the weight $u X_d + w + v X_d^{-1}$ .

• • In the region $\{(x,y) \mid y \le 1\}$ , the step set is $\{(-1,-1),(-2,-2),(-2,-1)\}$ . Steps of type $(-1,-1)$ are equipped with the weight $-w$ , while steps of type $(-2,-2)$ are equipped with the weight $-u v$ .

The paths are nonintersecting in the region $\{(x,y) \mid y \ge 1\}$ . In the region $\{(x,y) \mid y \le 1\}$ , there may be intersections, however, no two steps of different paths may have a common end point.

The weight of a family is $\prod _{i=1}^{n} X_i^{n-1}$ multiplied by the product of the weights of all its steps where the weight of a step is $1$ if it has not been specified. Let $\sigma $ be the permutation so that the i-th starting point is connected to the $\sigma (i)$ -th end point, then the sign of the family is $\operatorname {\mathrm {sgn}} \sigma $ .

An example is provided in Figure 13. Its weight is

$$ \begin{align*} \operatorname{\mathrm{sgn}} \sigma (-u v)^3 (-w)^5 X_1^5 & X_2^5 X_3^5 X_4^5 X_5^5 X_6^5 (u X_1+ w+ v X_1^{-1}) (u X_2+ w+ v X_2^{-1})^2\\ & \times (u X_3+ w+ v X_3^{-1})^2 (u X_4+ w+ v X_4^{-1}) (u X_5+ w+ v X_5^{-1})^2 (u X_6 + v X_6^{-1})^2 \end{align*} $$

with the associated permutation $\sigma = (1\;2\;4\;3\;5\;6)$ .

Figure 13 Example of the lattice paths in Theorem 7.1 for $n=6$ . The paths are drawn in alternating colors for a better distinction.

Again we list all families of paths from Theorem 7.1 for the case $n=2$ along with their weights up to the overall factor $X_1 X_2$ .

Proof of Theorem 7.1.

In the determinant in (3.5), we add t times the $(n-j)$ -th column to the $(n-j+1)$ -st column, for $j=1,\ldots ,n-1$ , in this order. We repeat this for $j=1,\ldots ,n-2$ , then for $j=1,\ldots ,n-3$ , etc. We obtain

$$ \begin{align*} & \prod_{i=1}^{n} X_i^{n-2} (u-v X_i^{-2})^{-1}\\ & \times \frac{\det\limits_{1 \le i,j \le n} \left( \left(u^2 X_i^2 + u w X_i+t \right)^{j-1} \left(u^2 X_i^2 + u w X_i \right) - \left(v^2 X_i^{-2} + v w X_i^{-1}+t\right)^{j-1} \left(v^2 X_i^{-2} + v w X_i^{-1}\right) \right)}{\prod\limits_{1 \le i < j \le n} (X_j - X_i)\left( u - v X_i^{-1} X_j^{-1} \right)}. \end{align*} $$

Now, for $j=1$ , we can replace $\left (u^2 X_i^2 + u w X_i \right )$ and $\left (v^2 X_i^{-2} + v w X_i^{-1}\right )$ by $\left (u^2 X_i^2 + u w X_i+t \right )$ and $\left (v^2 X_i^{-2} + v w X_i^{-1}+t\right )$ , respectively, as the t cancels in this case. Then we add t times the j-th column to the $(j+1)$ -st column, for $j=1,2,\ldots ,n-1$ , in this order. The result is

$$ \begin{align*} \prod_{i=1}^{n} X_i^{n-2} (u-v X_i^{-2})^{-1} \frac{\det_{1 \le i,j \le n} \left( \left(u^2 X_i^2 + u w X_i+t \right)^{j} - \left(v^2 X_i^{-2} + v w X_i^{-1}+t\right)^{j} \right)}{\prod_{1 \le i < j \le n} (X_j - X_i)\left( u - v X_i^{-1} X_j^{-1} \right)}. \end{align*} $$

Setting $t=u v$ , we obtain after some further modifications

$$ \begin{align*} \prod_{i=1}^{n} X_i^{n-1} \frac{\det_{1 \le i,j \le n} \left( (u X_i + w + v X_i^{-1})^{j} \frac{u^j X_i^j- v^j X_i^{-j}}{u X_i - v X_i^{-1}} \right)}{\prod_{1 \le i < j \le n} (X_j - X_i)\left( u - v X_i^{-1} X_j^{-1} \right)}. \end{align*} $$

We will apply Lemma 4.1 with $Y_i=u X_i +w + v X_i^{-1}$ . We need to express the entries in the i-th row in terms of formal Laurent series in $Y_i$ . By Lemma 4.2, this entry is

$$ \begin{align*} & (u X_i + w + v X_i^{-1})^{j} \sum_{r=0}^{(j-1)/2} (-uv)^r \binom{j-r-1}{r} (u X_i + v X_i^{-1})^{j-2r-1} \\ & =(u X_i + w + v X_i^{-1})^{j} \sum_{r=0}^{(j-1)/2} (-uv)^r \binom{j-r-1}{r} (u X_i + v X_i^{-1}+w-w)^{j-2r-1} \\ & =\sum_{r=0}^{(j-1)/2} \sum_{s \ge 0} (-uv)^r (-w)^{j-2r-1-s} \binom{j-r-1}{r} \binom{j-2r-1}{s} (u X_i +w + v X_i^{-1})^{j+s}. \end{align*} $$

Using Lemma 4.1 and setting $s=t-j-1$ , the expression in (3.5) is equal to

(7.1) $$ \begin{align} \prod_{i=1}^n X_i^{n-1} \det_{1 \le i,j \le n} \left( \sum_{t\in\mathbb{Z}} c_{t,j}(u,v,w) h_{t-i}(u X_1 + w + v X_1^{-1},\ldots,u X_i + w + v X_i^{-1}) \right), \end{align} $$

with $c_{t,j}(u,v,w)$ defined as

(7.2) $$ \begin{align} \sum_{r=0}^{(j-1)/2} (-uv)^r (-w)^{2j-2r-t} \binom{j-r-1}{r} & \binom{j-2r-1}{t-j-1} \nonumber\\ & = \sum_{r=0}^{(j-1)/2} (-uv)^r (-w)^{2j-2r-t} \binom{j-r-1}{r,2j-2r-t,t-j-1}. \end{align} $$

We claim that $c_{t,j}(u,v,w)$ is the generating function of lattice paths from $(0,0)$ to $(t-2,j-1)$ with step set $\{(1,1),(2,2),(2,1)\}$ with respect to the weight

$$ \begin{align*} (-uv)^{\# (2,2)\text{-steps}}(-w)^{\# (1,1)\text{-steps}}. \end{align*} $$

Indeed, suppose that there are r steps of type $(2,2)$ , x steps of type $(1,1)$ and y steps of type $(2,1)$ . Then

$$ \begin{align*} r \cdot (2,2) + x \cdot (1,1) + y \cdot (2,1) = (t-2,j-1), \end{align*} $$

which implies $x=2j-2r-t$ and $y=t-j-1$ . Therefore, the number of such paths is indeed $\binom {j-r-1}{r,2j-2r-t,t-j-1}$ . From Lemma 4.3, it follows that the matrix underlying the determinant in (7.1) has the following combinatorial interpretation, see Figure 13: We consider lattice paths from $(1,1),(2,2),\ldots ,(n,n)$ to $(2,1),(2,0),\ldots ,(2,-n+2)$ as follows:

• • In the region $\{(x,y) \mid y \ge 1\}$ , the step set is $\{(1,0),(0,-1)\}$ , and horizontal steps with distance d from the x-axis are equipped with the weight $u X_d + w + v X_d^{-1}$ . Suppose $(i,i)$ is the starting point and $(t,1)$ is last lattice point in this region, then

  $$ \begin{align*} h_{t-i}(u X_1 + w + v X_1^{-1},\ldots,u X_i + w + v X_i^{-1}) \end{align*} $$
  is the generating function of such paths.

• • In the region $\{(x,y) \mid y \le 1\}$ , the step set is $\{(-1,-1),(-2,-2),(-2,-1)\}$ . Steps of type $(-1,-1)$ are equipped with the weight $-w$ , while steps of type $(-2,-2)$ are equipped with the weight $-u v$ . Suppose a path goes from $(t,1)$ to $(2,-j+2)$ , then the generating function of such paths is $c_{t,j}(u,v,w)$ .

• • Paths starting in $(i,i)$ are equipped with the additional weight $X_i^{n-1}$ .

The paths are nonintersecting in the region $\{(x,y) \mid y \ge 1\}$ . In the region $\{(x,y) \mid y \le 1\}$ , there may be intersections, however, no two steps of different paths may have a common end point. This concludes the proof of Theorem 7.1.

Towards a signless version of Theorem 7.1?

It seems less clear how to show that the generating function of lattice paths from Theorem 7.1 has no negative coefficients. It is tempting to proceed as in the case of the first interpretation since Lemma 5.1 does have the following counterpart (Lemma 7.2). However, it is unclear what replaces the Gessel–Viennot involution from the previous section.

Lemma 7.2. Let $i,j$ be positive integers, then

$$ \begin{align*} \sum_{t\in\mathbb{Z}} c_{t,j}(u,v,w) h_{t-i}(u X_1 + w + v X_1^{-1},\ldots,u X_i + w + v X_i^{-1}) \end{align*} $$

is the generating function of lattice paths from $(i,i)$ to $(2j,1)$ using the step set $\{(1,0), (0,-1)\}$ , with three types of $(1,0)$ -steps, colored in red, blue, and green, and with weights $u X_d$ , $v X_d^{-1}$ , and w, respectively, where d is the distance from the x-axis and $c_{t,j}(u,v,w)$ is given as in (7.2), and right of the line $y=x-j$ there is no green horizontal step and also no pair of consecutive steps where the first is blue and the second is red.

Proof. Recall that the sum in the lemma is the generating function of lattice paths from $(i,i)$ to $(2,-j+2)$ using the step set and weights as given in the lemma in the region $\{(x,y) \mid y \ge 1\}$ , while below $y=1$ , the step set is $\{(-1,-1),(-2,-2),(-2,-1)\}$ , where the steps of type $(-1,-1)$ and $(-2,-2)$ are equipped with the weight $-w$ and $-u v$ , respectively.

We will construct a sign-reversing involution on such paths where one of the following is satisfied:

• • The path contains a step of type $(-1,-1)$ .

• • The path contains a step of type $(-2,-2)$ .

• • Right of the line $y=x-j$ , the path contains a green horizontal step.

• • Right of the line $y=x-j$ , the path contains a pair of consecutive horizontal steps where the first is blue and the second is red.

A path that does not fall into that class has only steps of type $(-2,-1)$ below the line $y=1$ , and, since its end point is $(2,-j+2)$ , the last point on the line $y=1$ is $(2j,1)$ . Consequently, we obtain a path as described in the lemma after deleting the (fixed) portion below the line $y=1$ .

In order to construct the sign-reversing involution, we observe the following: Assuming that $(t,1)$ is the last point on the line $y=1$ , the number of $(-2,-1)$ -steps is $t-j-1$ , while the total number of steps from $(i,i)$ to $(t,1)$ is $t-1$ . Let r be the maximal number of consecutive $(-2,-1)$ -steps right after $(t,1)$ . Now consider the r steps before $(t,1)$ is reached. Since $r \le t-j-1 < t-1$ , these steps exist.

Among these, we consider occurrences of green horizontal steps and occurrences of pairs of consecutive horizontals steps where the first is blue and the second is red, and, if there is such an occurrence, we consider the one that is closest to $(t,1)$ . Suppose it is a green horizontal step at position s before $(t,1)$ , then we delete this step and replace the s-th $(-2,-1)$ -step after $(t,1)$ by a $(-1,-1)$ -step. Clearly, this shifts the last point on the line $y=1$ from $(t,1)$ to $(t-1,1)$ . If we have a pair of consecutive horizontal steps where the first is blue and the second is red, and they are in position $s,s+1$ before $(t,1)$ , then we delete the blue and the red step and replace the two $(-2,-1)$ -steps at positions s and $s+1$ after $(t,1)$ by $(-2,-2)$ . In this case, the last point on the line $y=1$ is shifted by $2$ to the left.

If there is neither a green horizontal step nor a pair of consecutive horizontal steps where the first is blue and the second is red, then there has to be either a $(-1,-1)$ -step or a $(-2,-2)$ -step after the r-th $(-2,-1)$ -step. In the first case, we replace it by a $(-2,-1)$ -step and insert a green horizontal step in position $r+1$ before $(t,1)$ . This shifts the last point on the line $y=1$ from $(t,1)$ to $(t+1,1)$ . In the second case, we replace the $(-2,-2)$ -step by two $(-2,-1)$ -steps and insert a pair of consecutive horizontal steps where the first is blue and the second is red in positions $r+2$ and $r+1$ before $(t,1)$ , which shifts the last point on the line $y=1$ from $(t,1)$ to $(t+2,1)$ .

This involution is clearly sign-reversing.

7.2 The third type of families of lattice paths

Theorem 7.3. For $n \ge 1$ , the generating function of arrowed monotone triangles with bottom row $0,2,\ldots ,2n-2$ is equal to the generating function of families of n nonintersecting lattice paths with starting points in $A_i=\{(n-i+1,2n),(n+i-1,2n)\}$ , $i=1,2,\ldots ,n$ , to $E_j=(j,-j+1)$ , $j=1,2,\ldots ,n$ with the following properties:

• • In the region $\{(x,y) \mid y \ge 1\}$ , the step set is $\{(1,0),(0,-1)\}$ . Horizontal steps at height $1,2,3,4,\ldots $ above the x-axis have weight $u X_1, v X_1^{-1}, u X_2, v X_2^{-1},\ldots $ , respectively.

• • In the region $\{(x,y) \mid y \le 1\}$ , the step set is $\{(-1,-1),(0,-1)\}$ , where steps of type $(0,-1)$ are equipped with the weight w.

Let $1 < i_1 < i_2 < \ldots < i_k \le n$ be precisely the indices for which we choose $(n+i_k-1,2n) \in A_{i_k}$ as starting points (and not $(n-i_k+1,2n) \in A_{i_k}$ ). The weight of a family is

$$ \begin{align*} \prod_{l=1}^k (-uv)^{i_l-1} \prod_{i=1}^n X_i^{n-1} \end{align*} $$

multiplied by the product of the weights of all its steps where the weight of a step is $1$ if it has not been specified.

Note that $A_1=\{(n,2n)\}$ consists of a single point. Note as well that in this interpretation, the sign has been incorporated into the weight. An example for the case $n=4$ is provided in Figure 14. For this interpretation, we omit to showcase the example for $n=2$ since it already involves $57$ configurations.

Figure 14 Example of the lattice paths in Theorem 7.3 for $n=4$ . The weight of this family of nonintersecting lattice paths is $(-uv)^{3-1} w^2 (u X_1) (u X_2) (u X_{3}) (v X_{3}^{-1}) X_1^3 X_2^3 X_3^3 X_4^3$ , which is equal to $u^5 v^3 w^2 X_1^4 X_2^4 X_3^3 X_4^3$ .

Proof of Theorem 7.3.

We interpret the Jacobi–Trudi-type expression (A.6) derived in Appendix A in terms of the following families of n nonintersecting lattice paths: We consider lattice paths from $A_i=\{(n-i+1,2n),(n+i-1,2n)\}$ to $E_j=(j,-j+1)$ for $1 \le i,j \le n$ , see Figure 14. The step set as well as the edge weights depend on whether or not we are below the line $y=1$ .

• • Above and on the line $y=1$ , the step set is $\{(1,0),(0,-1)\}$ . Horizontal steps at height d above the x-axis have weight $u X_d$ if $d \le n$ and the weight $v X_{d-n}^{-1}$ if $d>n$ . Assuming that $(k,1)$ is the last lattice point on the line $y=1$ , the generating function of lattice paths from $(n-i+1,2n)$ to $(k,1)$ is

  $$ \begin{align*} h_{k+i-n-1}(u X_1,\ldots,u X_n,v X_1^{-1},\ldots,v X_n^{-1}), \end{align*} $$
  while the generating function of those lattice paths from $(n+i-1,2n)$ to $(k,1)$ is
  $$ \begin{align*} h_{k-i-n+1}(u X_1,\ldots,u X_n,v X_1^{-1},\ldots,v X_n^{-1}). \end{align*} $$

  Paths that start in $(n+i-1,2n)$ get an additional weight $(-uv)^{i-1}$ , where the sign will be explained below.

• • Below the line $y=1$ , the step set is $\{(-1,-1),(0,-1)\}$ , and since we want to reach $(j,-j+1)$ , there are $k-j$ steps of type $(-1,-1)$ and $2j-k$ steps of type $(0,-1)$ , which gives in total $\binom {j}{k-j}$ choices. The latter steps carry the weight w.

• • We have to multiply with an overall factor of $\prod _{i=1}^{n} X_i^{n-1}$ .

Note that the factor $\frac {1}{2}$ is taken into account by the fact that $A_1$ consists of a single point.

The nonintersecting property does not force that $A_i$ is connected to $E_i$ by a path, so we still need to take the sign into account: Suppose we have the starting points $(n-i+1,2n)$ precisely for $n \ge i_p> i_{p-1} > \ldots > i_1=1$ and starting points $(n+i-1,2n)$ for $1 < j_1 < j_2 < \ldots < j_{n-p} \le n$ . Then the only permutation for which the n-tuple of paths can be nonintersecting is

$$ \begin{align*} (i_p,i_{p-1},\ldots,i_1,j_1,\ldots,j_{n-p})^{-1}, \end{align*} $$

the sign of which is $(-1)^{(i_1-1)+(i_2-1)+\cdots +(i_p-1)}$ . Thus, taking the factor $(-1)^{\binom {n}{2}}$ into account, each path with $(n+i-1,2n)$ as starting point contributes $(-1)^{i-1}$ to the sign.