Coxeter and Schubert combinatorics of μ \mu –Involutions Jack Chen-An Chou, Zachary Hamaker Abstract . The variety of complete quadrics is the wonderful compactification of G ​ L n / O n GL_{n}/O_{n} and admits a cell decomposition into Borel orbits indexed by combinatorial objects called μ \mu –involutions. We study Coxeter–theoretic properties of μ \mu –involutions with results including a combinatorial description for their atoms, an exchange lemma, and transposition-like operators that characterize their Bruhat order. The corresponding orbit closures can be realized inside the flag variety. In this setting, we study the cohomology representatives of these orbits, which are, up to a scalar, the μ \mu –involution Schubert polynomials. We expand μ \mu –involution Schubert polynomials as a multiplicity-free sum of ν \nu –involution Schubert polynomials when ν \nu refines μ \mu and provide recurrences analogous to Monk’s rule for Schubert polynomials. 1. Introduction The complete flag variety \text{FL}_{n}=GL_{n}/B has a natural stratification given by Borel orbit closures, indexed by permutations, whose classes form a basis for its cohomology ring. This ring is the modern setting for Schubert calculus, an area of algebraic geometry that studies enumerative aspects of intersection theory. Schubert polynomials, introduced by Lascoux and Schützenberger [ 20 ] , represent the cohomology classes of these orbit closures and provide a combinatorial model for this theory. Their rich structure can be seen through the lens of geometry, representation theory, and combinatorics. Significant effort has been devoted to extending Schubert calculus beyond the setting of the Schubert stratification for \text{Fl}_{n} . One important such extension is the symmetric space given by orthogonal group orbit closures on \text{Fl}_{n} , which are indexed by involutions in the symmetric group. These were initially studied by Richardson and Springer [ 23 , 24 ] , with polynomial representatives introduced by Wyser and Yong in [ 28 ] . These representatives, divided by a predictable power of 2, are the involution Schubert polynomials . A general theorem of Brion [ 5 ] shows involution Schubert polynomials are multiplicity-free sums of Schubert polynomials. The terms in this sum, which we call atoms , have a combinatorial description due to Can, Joyce and Wyser [ 6 ] (see also [ 13 ] ). Hamaker, Marberg, and Pawlowski [ 15 ] demonstrate a recurrence relation on involution Schubert polynomials analogous to Monk’s rule. Orthogonal orbits of \text{Fl}_{n} can be viewed as Borel orbits on GL_{n}/O_{n} . This latter space is not compact, but has a wonderful compactification as the variety of complete quadrics \text{Q}_{n} [ 9 ] . In this paper, we study the Borel orbits of \text{Q}_{n} and their realizations as subvarieties of \text{Fl}_{n} . Passing from GL_{n}/O_{n} to \text{Q}_{n} introduces boundary strata corresponding to partial degenerations of a quadric, and these strata are indexed by compositions of n . In the stratum indexed by \mu=(\mu_{1},\dots,\mu_{k}) the Borel orbits correspond to H_{\mu} orbits of \text{Fl}_{n} where H_{\mu} is the semidirect product of O_{\mu_{1}}\times\dots\times O_{\mu_{k}} and the parabolic subgroup P_{\mu}\subseteq GL_{n} . These orbits are indexed by \mu –involutions , which are permutations \pi expressible as a concatenation of blocks B_{1},\dots,B_{k} with |B_{i}|=\mu_{i} such that each B_{i} has the relative order of an involution. Since H_{\mu} interpolates between O_{n} when \mu=(n) and B when \mu=(1^{n}) , \mu –involutions interpolate between involutions and permutations. Orbit closure containment defines a Bruhat order on \mu –involutions as \mu varies, which is a special case of a more general family of orders studied in [ 26 ] . For fixed \mu , there is also a weak order studied in [ 8 ] , which determines analogues of reduced expressions for \mu –involutions. In [ 1 ] , the authors show Bruhat order on \mu –involutions is determined by subword containment (see Theorems 2.10 and 2.11 ). Using this realization, we provide a novel characterization of Bruhat order on \mu –involutions using transposition-like operators t^{\mu}_{ab} (see Section 3.3 and specifically Theorem 3.11 ). The Brion results extend to this setting, so the classes of H_{\mu} orbit closures in \text{Fl}_{n} can be represented as sums of Schubert polynomials. Again rescaling by a predictable power of 2, for each \mu –involution \pi we obtain the \mu –involution Schubert polynomial \mathfrak{S}^{\mu}_{\pi} as a multiplicity-free sum of Schubert polynomials. The set of atoms in this expansion is denoted \mathcal{A}_{\mu}(\pi) . These polynomials can also be defined using divided difference operators applied to \mathfrak{S}^{\mu}_{w_{0}} , which is computed geometrically as a product of monomials and binomials in [ 7 ] or as a sum over \mathcal{A}_{\mu}(w_{0}) in [ 8 ] . See [ 18 ] for a summary of these results. Our first main result is a combinatorial description of \mathcal{A}_{\mu}(\pi) , generalizing the description of \mathcal{A}_{\mu}(w_{0}) from [ 8 ] . The definition of atoms for involutions can be extended to blocks, with \mathcal{A}(B) the set of block atoms for B (see Section 3.1 for details). The description is more easily stated in terms of inverse atoms, the set of which is \mathcal{A}_{\mu}^{-1}(\pi) . Theorem 1.1 (= Theorem 3.1 ) . For \pi a \mu –involution with blocks B_{1},\dots,B_{k} , \mathcal{A}_{\mu}^{-1}(\pi)=\{Q_{1}Q_{2}\dots Q_{k}:Q_{i}\in\mathcal{A}^{-1}(B_{i})\text{ for }i\in[k]\}. where Q_{1}Q_{2}\dots Q_{k} is the concatenation of Q_{1},Q_{2},\dots,Q_{k} . Our proof of Theorem 1.1 adapts the proof from [ 8 ] to this more general setting. As an application of this theorem, we expand \mu –involution Schubert polynomials as a multiplicity-free sum of \nu –involution Schubert polynomials when \nu refines \mu (see Theorem 3.7 ). This result generalizes Theorem 1.1 and allows for an improved description of Bruhat order for \mu –involutions when \mu varies (see Corollary 3.6 ). The other main result of this paper is a family of transition equations for \mu –involution Schubert polynomials, which is closely related to Monk’s rule for Schubert polynomials. The formula relies on the transposition operators t_{ab}^{\mu} we define that describe the strong Bruhat cover relations of \mu –involutions. Theorem 1.2 (= Theorem 3.18 ) . Let \tau be a \mu involution and (i\geq j) a cycle in one of its blocks, \displaystyle 2^{-\delta(i,j)}(x_{i}+x_{j})\mathfrak{S}_{\tau}^{\mu}=\sum_{\sigma\in\Phi(\tau;i,j)}\mathfrak{S}_{\sigma}^{\mu}-\sum_{\sigma\in\Psi(\tau;i,j)}\mathfrak{S}_{\sigma}^{\mu}\,, where \displaystyle\Phi(\tau;i,j)=\{t^{\mu}_{ab}(\tau)\neq\tau\,:\,a\in\{i,j\}\},\quad\Psi(\tau;i,j)=\{t^{\mu}_{ab}(\tau)\neq\tau\,:\,b\in\{i,j\}\}. A key technical tool in our proof of Theorem 1.2 is a novel exchange lemma like result for reduced expressions of \mu –involutions (see Theorem 3.16 ) generalizing an analogous result for involutions from [ 15 ] . 2. Background We first introduce some notation. Let [n]=\{1,2,\dots,n\} . For the rest of the paper, we will use w,u,v to denote permutations, y,z to denote involutions, \pi,\tau,\sigma to denote \mu -involutions, and \mu,\nu to denote compositions. Recall \mu=(\mu_{1},\dots,\mu_{k}) is a composition of n with length \ell(\mu)=k , denoted \mu\vDash n , if \sum_{i=1}^{k}\mu_{i}=n and \mu_{i}\in[n] for all i . For \mu,\nu\vDash n , we say \mu refines \nu if \left\{\sum_{i=1}^{j}\mu_{i}:j\in[\ell(\mu)]\right\}\supseteq\left\{\sum_{i=1}^{j}\nu_{i}:j\in[\ell(\nu)]\right\}. For example, (3,2,3) refines (5,3) , (3,5) and (8) . 2.1. Permutations For A\subseteq\mathbb{N} finite, let S_{A} be the set of permutations of A and S_{n}=S_{[n]} . We typically write permutations in one-line notation— for Q\in S_{A} write Q=Q(1)\dots Q(k) with Q(i)\in A distinct for each i\in[k] . The standardization of Q is the permutation \mathrm{st}(Q)\in S_{|A|} whose one-line notation has the same relative order as Q . For Q\in S_{A} and a\in A , let Q^{-1}(a) be the index i so that Q(i)=a . Then the relative value of a in Q is r_{Q}(a)=\mathrm{st}(Q)(Q^{-1}(a)) . For example, with Q=63274 we have \mathrm{st}(Q)=42153 so Q^{-1}(4)=5 and r_{Q}(4)=3 . When n is clear, let e=12\dots n and w_{0}=n\dots 21 be the identity and reverse permutations in S_{n} . Let t_{ij} denote the transposition (i,j) and s_{i}=(i,i{+}1) . Recall for w\in S_{n} that wt_{ij} is the same as w except that w(i) and w(j) are swapped, while t_{ij}w and w are the same except the positions containing i and j are swapped. For example, with w=4231 we have wt_{13}=3241 and t_{13}w=4213 . Viewing permutations as functions, S_{n} is a Coxeter group with presentation S_{n}=\langle s_{1},\dots,s_{n-1}\mid s_{i}^{2}=1,s_{i}s_{j}=s_{j}s_{i}\ \mathrm{for}\ |i-j|>1,\ s_{i}s_{i+1}s_{i}=s_{i+1}s_{i}s_{i+1}\ \mathrm{for}\ i\in[n-2]\rangle. For w\in S_{n} , an expression w=s_{a_{1}}\dots s_{a_{p}} is reduced if it is of minimal length with associated reduced word \textbf{a}=(a_{1},\dots,a_{p}) . The length \ell(w) is then the length of a reduced word for w . Let \mathcal{R}(w) be the set of reduced words for w . Expressions for w satisfy: Lemma 2.1 (Deletion Property, [ 4 ] *Prop. 1.4.7) . For w=s_{a_{1}}\dots s_{a_{p}}\in S_{n} so that the word (a_{1},\dots,a_{p}) is not reduced, there exists k,m\in[p] so that w=s_{a_{1}}\dots\widehat{s_{a_{k}}}\dots\widehat{s_{a_{m}}}\dots s_{a_{p}} . We make extensive use of two partial orders on S_{n} . The first is (strong) Bruhat order (S_{n},\leq) , where v\leq w if every \textbf{a}\in\mathcal{R}(w) has a subword in \mathcal{R}(v) . The second is weak (Bruhat) order (S_{n},\leq_{W}) where v\leq w if some \textbf{a}\in\mathcal{R}(w) has a prefix in \mathcal{R}(v) . Both partial orders are graded with rank function \ell . The cover relations for these partial orders are v\lessdot w if w=vt_{ij} and \ell(w)=\ell(v)+1 and v\lessdot_{W}w if w=vs_{i}>w for some i\in[n{-}1] . A parabolic subgroup of S_{n} is one generated by a subset of the simple transpositions. In particular, for \mu=(\mu_{1},\dots,\mu_{k})\vDash n we have the parabolic subgroup S_{\mu}=S_{\mu_{1}}\times\dots\times S_{\mu_{k}} ; here we identify S_{\mu_{i}} with \langle s_{m_{i}+1},\dots,s_{m_{i}+\mu_{i}-1}\rangle where m_{i}=\mu_{1}+\dots+\mu_{i-1} (so m_{1}=0) . The parabolic quotient S^{\mu} is the set of minimal length coset representatives for S_{\mu} . We have: Proposition 2.2 ( [ 4 ] *Prop. 2.4.4) . For \mu\vDash n and w\in S_{n} there exist unique u\in S_{\mu} , v\in S^{\mu} so that w=uv . Furthermore \ell(w)=\ell(u)+\ell(v) . The 0–Hecke monoid is the unique monoid (S_{n},\circ) satisfying w\circ s_{i}=\begin{cases}ws_{i}&ws_{i}>w,\\ w&ws_{i}w(k)>w(j) . (3) If (i,j),(k,l)\in\mathrm{Cyc}(y) with iw(i)\geq w(j) . For y\in\mathcal{I}_{n} , Theorem 2.3 is perhaps more easily understood in terms of the inverse atoms \mathcal{A}^{-1}(y)=\{w^{-1}:w\in\mathcal{A}(y)\} . Here, for v\in\mathcal{A}^{-1}(y) and (i,j),(k,l)\in\mathrm{Cyc}(y) with i>j , • (1) says that i occurs before j in v ; • from (2), when (i,j) and (k,l) nest with jq\,\} \displaystyle\Phi^{-}(y,p)=\{\,z\in\mathcal{I}_{\mathbb{Z}}:y\lessdot_{I}z\text{ and }z=t_{ip}^{\mathcal{I}}(y)\text{ for an integer }ik\geq l ). One can check that the fourth column contains exactly the involutions coming from the first column restricted to the values \{i,j,k,l\} in B_{L} . A completely symmetric argument can be made to show that Q(k+1)\cdots Q(m) is an inverse block atom for the block involution obtained from B ’s restriction to Q(k{+1})\dots Q(m) . Therefore, the claim follows. ∎ Corollary 3.4 . If \nu refines \mu , then any inverse atom w in \mathcal{I}_{\mu} is also an inverse atom in \mathcal{I}_{\nu} . Proof. By Theorem 3.1 , every inverse atom is a concatenation of its block inverse atoms, so the statement follows from repeated application of Lemma 3.3 . ∎ Lemma 3.5 . Let \pi\in\mathcal{I}_{\mu} and \tau\in\mathcal{I}_{\nu} where \nu refines \mu . Then either \mathcal{A}_{\nu}^{-1}(\tau)\subseteq\mathcal{A}_{\mu}^{-1}(\pi) or \mathcal{A}_{\nu}^{-1}(\tau)\cap\mathcal{A}_{\mu}^{-1}(\pi)=\varnothing . Proof. Suppose w\in\mathcal{A}_{\nu}^{-1}(\tau)\cap\mathcal{A}_{\mu}^{-1}(\pi) . Then for any w^{\prime}\in\mathcal{A}_{\nu}^{-1}(\tau) , Theorem 3.1 and Theorem 2.4 (applied to the standardization of block inverse atoms) imply w^{\prime} can be obtained from w through a series of transformations from ( 1 ) in the blocks of \nu . Since \nu refines \mu , each such equivalence occurs in a block of \mu , so w^{\prime}\in\mathcal{A}_{\mu}^{-1}(\pi) , hence \mathcal{A}_{\nu}^{-1}(\tau)\subseteq\mathcal{A}_{\mu}^{-1}(\pi) . ∎ A restatement of Lemma 3.5 in terms of Bruhat order is: Corollary 3.6 . For \pi\in\mathcal{I}_{\mu} and \tau\in\mathcal{I}_{\nu} where \nu refines \mu and \ell(\nu)=\ell(\mu)+1 , we have \tau\lessdot^{\mu}\pi if some w\in\mathcal{A}_{\nu}(\tau) is also in \mathcal{A}_{\mu}(\pi) . We are now prepared to prove the main result of this subsection. Theorem 3.7 . Let \pi\in\mathcal{I}_{\mu} , and suppose \nu refines \mu , then \displaystyle\mathfrak{S}_{\pi}^{\mu}=\sum_{\tau\in\mathcal{I}_{\nu},\,\tau\leq_{\mathcal{A}}\pi}\mathfrak{S}_{\tau}^{\nu} Proof. We show by double containment of inverse atoms. Let S=\bigsqcup_{\tau\in\mathcal{I}_{\nu},\,\tau\leq_{\mathcal{A}}\pi}\mathcal{A}_{\nu}^{-1}(\tau). By definition of \leq_{\mathcal{A}} we see S is a subset of \mathcal{A}_{\mu}^{-1}(\pi) . For any w\in\mathcal{A}_{\mu}^{-1}(\pi) , w is also an inverse atom of \tau for some \tau\in\mathcal{I}_{\nu} by Corollary 3.4 . Then by Corollary 3.6 \mathcal{A}_{\nu}^{-1}(\tau) is a subset of \mathcal{A}_{\mu}^{-1}(\pi) , so \tau\leq_{\mathcal{A}}\pi , hence w\in S . Therefore \mathcal{A}_{\mu}^{-1}(\pi) is a subset of S . ∎ Example 3.8 . Let \pi=75421|63 , so \mu=(5,2) . Then for any composition \nu that refines (5,2) , we can expand \mathfrak{S}_{\pi}^{\mu} into a positive and multiplicity-free sum of \nu -involution Schubert polynomials. For example, suppose that \nu=(2,3,1,1) . The inverse atoms of \pi \displaystyle\mathcal{A}_{\mu}^{-1}(\pi)=\{4527163,5247163,5271463,4571263,5712463,4715263,7145263,7152463\} are all inverse atoms of some \nu -involution from the following set \displaystyle B:=\{45|721|6|3,52|741|6|3,57|124|6|3,47|152|6|3,71|542|6|3\}\subseteq\mathcal{I}_{\nu}. By Lemma 3.5 , B is also exactly the set of \nu -involutions \tau such that \tau\leq_{\mathcal{A}}\pi . Therefore, by Theorem 3.7 , \displaystyle\mathfrak{S}_{75421|63}^{\mu}=\mathfrak{S}_{45|721|6|3}^{\nu}+\mathfrak{S}_{52|741|6|3}^{\nu}+\mathfrak{S}_{57|124|6|3}^{\nu}+\mathfrak{S}_{47|152|6|3}^{\nu}+\mathfrak{S}_{71|542|6|3}^{\nu}\,. ∎ 3.3. Transpositions for \mu –Bruhat order We first introduce notation for covers in \mu –Bruhat order. Let \pi and \tau be \mu –involutions so that \pi\lessdot^{\mu}\tau . By Theorem 2.10 , for each w\in\mathcal{A}_{\mu}(\tau) there exists v\in\mathcal{A}_{\mu}(\pi) so that v\lessdot w . Necessarily, w=vt_{ij} for some i,j . We define transposition like operators on \mu –involutions using this relationship. Definition 3.9 . For \pi a \mu –involution, if there exists v\in\mathcal{A}_{\mu}(\pi) and \tau a \mu –involution so that v\lessdot vt_{ij} and vt_{ij}\in\mathcal{A}_{\mu}(\tau) we define t_{ij}^{\mu}(\pi)=\tau . Otherwise, define t_{ij}^{\mu}(\pi)=\pi . ∎ A priori, it is unclear that the operators t_{ij}^{\mu} are well-defined. It could be the case that there exist atoms u,v\in\mathcal{A}_{\mu}(\pi) so that u\lessdot ut_{ij} and v\lessdot vt_{ij} where ut_{ij} and vt_{ij} are atoms for different \mu –involutions. We show that this cannot occur. Lemma 3.10 . For \mu\vDash n and i,j\in[n] , each operator t_{ij}^{\mu}:\mathcal{I}_{\mu}\to\mathcal{I}_{\mu} is well-defined. Proof. Let \pi\in\mathcal{I}_{\mu} and v,v^{\prime}\in\mathcal{A}_{\mu}^{-1}(\pi) so that t_{ij}v and t_{ij}v^{\prime} are also inverse \mu –atoms, say for \sigma,\tau\in\mathcal{I}_{\mu} , respectively. The result will follow by showing \mathrm{Cyc}_{\mu}(\sigma)=\mathrm{Cyc}_{\mu}(\tau) , which we detect by applying the characterization for \mu –atoms in Theorem 3.1 to t_{ij}v and t_{ij}v^{\prime} . Blocks that do not contain i or j are unchanged, so \sigma,\tau agree on these blocks. We now consider the blocks containing i and j . If the values i,j are in the same block B , the result follows from Theorem 2.6 since v|_{B} and v^{\prime}|_{B} are inverse block atoms for the same block involution. Now let i be in block B_{1} and j be in block B_{2} of \pi . Consider c_{1},c_{2}\in\mathrm{Cyc}_{B_{1}}(\pi) . In v and v^{\prime} , the possible subwords of the values in these cycles is determined by Theorem 2.3 . Since t_{ij}v and t_{ij}v^{\prime} are inverse \mu –atoms, the subwords for these same values correspond to pairwise cycle structures in their associated \mu –involutions. We must show the resulting pairwise cycle structures are the same. When c_{1},c_{2} do not contain i , their corresponding subwords are unchanged. Therefore, we can reduce to the case where c_{1}=(i,h)/(h,i) and c_{2}=(k,l) . Further, when c_{1} and c_{2} do not nest the values h,i,k,l form a unique subword in v and v^{\prime} , so t_{ij}v and t_{ij}v^{\prime} have the same subword for values h,j,k,l . Therefore, the cycles containing these values in \sigma and \tau must be the same. The possible cases where c_{1} and c_{2} nest are treated in Figure 4 . Since v\lessdot t_{ij}v , only certain subwords can appear in the third column. Note that all the subwords in the fourth column correspond to the unique cycle structure in the fifth column. By taking the reverse complement, the same argument holds for B_{2} so result follows. ∎ Relative order \pi –cycles \mathcal{A}_{\mu}^{-1}(\pi) –subwords \mathcal{A}_{\mu}^{-1}(\tau) –subwords \tau –cycles lv(k) so wi>j , r cannot appear to the left of i,j , so rijs cannot be a subword in any inverse atom, a contradiction. A similar argument holds for C . By Theorem 2.14 , the left hand side expands as \displaystyle 2^{-\delta(i,j)}(x_{i}+x_{j})\mathfrak{S}_{\tau}^{\mu} \displaystyle=2^{-\delta(i,j)}\sum_{v\in\mathcal{A}_{\mu}(\tau)}(x_{i}+x_{j})\mathfrak{S}_{v}=\sum_{u\in A}\mathfrak{S}_{u}-\sum_{u\in C}\mathfrak{S}_{u}. Now define A^{\prime}:=\{u\in\mathcal{A}_{\mu}(\sigma):\sigma\in\Phi(\tau;i,j)\} and C^{\prime}:=\{w\in\mathcal{A}_{\mu}(\sigma):\sigma\in\Psi(\tau;i,j)\} . Note A^{\prime}\subseteq A and C^{\prime}\subseteq C by Theorem 3.11 . Then the right hand side is \sum_{u\in A^{\prime}}\mathfrak{S}_{u}-\sum_{w\in C^{\prime}}\mathfrak{S}_{w}. Then the result will follow by showing A\setminus A^{\prime}=C\setminus C^{\prime} . Let w\in A\setminus A^{\prime} . By the definition of A , there exists v\in\mathcal{A}_{\mu}(\tau) so that w=vt_{ab} with a