Characterizing finite solvable groups through the nilpotency probability Andrea Lucchini Abstract . Given a finite group G G , we denote by ν ⁡ ( G ) \nu(G) the probability that two randomly chosen elements of G G generate a nilpotent subgroup. We prove that if ν ⁡ ( G ) > 1 / 12 , \nu(G)>1/12, then G G is solvable. 2020 Mathematics Subject Classification 20D60, 20P05 1. Introduction Given a finite group G , let \mathcal{N}(G) be the set of ordered pairs (g_{1},g_{2})\in G^{2} with the property that \langle g_{1},g_{2}\rangle is a nilpotent subgroup of G . Then \nu(G):=\frac{|\mathcal{N}(G)|}{|G|^{2}} expresses the probability that two randomly chosen elements of G generate a nilpotent subgroup. Guralnick and Wilson [ 4 ] proved that if \nu(G)>\nu(\perm(3))=1/2 , then G is nilpotent. However, the following question, raised in [ 5 ] , remains open: does \nu(G)>1/12 imply that G is solvable? In this note, we provide an affirmative answer. Theorem 1 . Let G be a finite group. If \nu(G)>\frac{1}{12}, then G is solvable. Since \nu(\mathrm{Alt}(5))=1/12, the previous result is sharp. 2. Proof of Theorem 1 Let us start this section with a couple of definitions. Definition 2 . Let G be a finite group, N a normal subgroup of G and g_{1},g_{2}\in G. We define \displaystyle\mathcal{N}_{g_{1},g_{2}}(G,N) \displaystyle:=\{(n_{1},n_{2})\in N^{2}\mid\langle n_{1}g_{1},n_{2}g_{2}\rangle\text{ is nilpotent }\}, \displaystyle\nu_{g_{1},g_{2}}(G,N) \displaystyle:=\frac{|\mathcal{N}_{g_{1},g_{2}}(G,N)|}{|N|^{2}}. Definition 3 . Let S be a finite nonabelian simple group. We identify S with the subgroup of its inner automorhisms and we define \tilde{\nu}(S)=\max_{a_{1},a_{2}\in\aut S}\nu_{a_{1},a_{2}}(\aut S,S). Proposition 4 . Assume that a finite group G contains a unique minimal normal subgroup, say N , and that N is nonabelian. Then \nu(G)\leq\tilde{\nu}(S), where S is a composition factor of N. Proof. Assume that N=S^{u}, with u\in\mathbb{N}. We may identify G with a subgroup of the wreath product \aut S\wr\perm(u). In this identification, any element of G can be written in the form (\alpha_{1},\dots,\alpha_{u})\rho, with \alpha_{1},\dots,\alpha_{u}\in\aut S and \rho\in\perm(u). Moreover, we identify S with the inner automorphism group \inn(S) and therefore N=\{(s_{1},\dots,s_{u})\mid s_{1},\dots,s_{u}\in S\}. Now let g_{1}=(a_{1},\dots,a_{u})\sigma,\quad g_{2}=(b_{1},\dots,b_{u})\tau\quad\in G. Write \sigma=\sigma_{1}\cdots\sigma_{c},\quad\tau=\tau_{1}\cdots\tau_{d}, as products of disjoint cyclic permutations and assume that 1 belongs to the supports of \sigma_{1} and \tau_{1}. Suppose \sigma_{1}=(1,i_{2},\dots,i_{e}),\quad\tau_{1}=(1,j_{2},\dots,j_{f}). Let X=\{(\alpha_{1},\dots,\alpha_{u})\rho\in G\mid\rho(1)=1\}. The map \phi\colon X\to\aut S sending (\alpha_{1},\dots,\alpha_{u})\rho to \alpha_{1} is a group homomorphism. Now let n_{1}=(s_{1},\dots,s_{u}),n_{2}=(t_{1},\dots,t_{u})\in N . Then (n_{1}g_{1})^{e},(n_{2}g_{2})^{f}\in X and \phi((n_{1}g_{1})^{e})=s_{1}a_{1}s_{i_{2}}a_{i_{2}}\cdots s_{i_{e}}a_{i_{e}},\quad\phi((n_{2}g_{2})^{f})=t_{1}b_{1}t_{j_{2}}b_{j_{2}}\cdots t_{j_{f}}b_{j_{f}}. Let \gamma_{1}=a_{1}s_{i_{2}}a_{i_{2}}\cdots s_{i_{e}}a_{i_{e}},\quad\gamma_{2}=b_{1}t_{j_{2}}b_{j_{2}}\cdots t_{j_{f}}b_{j_{f}}. If (s_{1},t_{1})\notin\mathcal{N}_{\gamma_{1},\gamma_{2}}(\aut S,S), then \phi(\langle(n_{1}g_{1})^{e},(n_{2}g_{2})^{f}\rangle)=\langle s_{1}\gamma_{1},t_{1}\gamma_{2}\rangle is not nilpotent. In particular \langle n_{1}g_{1},n_{2}g_{2}\rangle is not nilpotent. Thus, given s_{i},t_{j} for i\neq 1 and j\neq 1 , there are at least |S|^{2}(1-\nu_{\gamma_{1},\gamma_{2}}(\aut S,S))\geq|S|^{2}(1-\tilde{\nu}(S)) choices for s_{1} and t_{1} such that \langle n_{1}g_{1},n_{2}g_{2}\rangle is not nilpotent. It follows that there are at least |S|^{2u}(1-\tilde{\nu}(S)) pairs (n_{1},n_{2})\in N^{2} such that \langle n_{1}g_{1},n_{2}g_{2}\rangle is not nilpotent. This is equivalent to saying that \nu_{g_{1},g_{2}}(G,N)\leq\tilde{\nu}(S) for every (g_{1},g_{2})\in G^{2}. But then \displaystyle\nu(G)|G|^{2}= \displaystyle\sum_{(Ng_{1},Ng_{2})\in(G/N)^{2}}\nu_{g_{1},g_{2}}(G,N) \displaystyle\leq\sum_{(Ng_{1},Ng_{2})\in(G/N)^{2}}|N|^{2}\tilde{\nu}(S) \displaystyle\leq|G|^{2}\tilde{\nu}(S), and consequently \nu(G)\leq\tilde{\nu}(S). ∎ Proposition 5 . If S is a finite nonabelian simple group, then \tilde{\nu}(S)\leq\tilde{\nu}(\mathrm{Alt}(5))=\frac{1}{12}, with equality only if S=\mathrm{Alt}(5). Proof. Given a_{1},a_{2}\in\aut S , let \mathcal{P}_{a_{1},a_{2}}(S)=\{(s_{1},s_{2})\in S^{2}\mid S\leq\langle s_{1}a_{1},s_{2}a_{2}\rangle\} . We define \pi_{a_{1},a_{2}}(S):=\frac{|\mathcal{P}_{a_{1},a_{2}}(S)|}{|S|^{2}},\quad\tilde{\pi}(S)=\min_{a_{1},a_{2}\in\aut S}\pi_{a_{1},a_{2}}(S). Clearly if (s_{1},s_{2})\in\mathcal{P}_{a_{1},a_{2}}(S), then \langle s_{1}a_{1},s_{2}a_{2}\rangle is not nilpotent, and therefore (s_{1},s_{2})\notin\mathcal{N}_{a_{1},a_{2}}(\aut S,S). Thus, \pi_{a_{1},a_{2}}(S)\leq 1-\nu_{a_{1},a_{2}}(\aut(S),S) for every a_{1},a_{2}\in\aut S, and therefore \tilde{\nu}(S)\leq 1-\tilde{\pi}(S). Results of Dixon [ 1 ] , Kantor–Lubotzky [ 6 ] and Liebeck–Shalev [ls] establish that \tilde{\pi}(S)\to 1 as |S|\to\infty. Explicit lower bounds for \tilde{\pi}(S) are given in [ 9 ] , where in particular all nonabelian simple groups S with \tilde{\pi}(S)\leq 9/10 are listed. Since 1-9/10=1/10>1/12 , this is not enough for our purposes, but more detailed information can be found in [ 8 ] . In particular, combining [ 8 , Table 5.1, Theorem 6.01, Lemma 7.1.1] , it turns out that if S is not an alternating group, then either \tilde{\pi}(S)>\frac{11}{12} (and consequently \tilde{\nu}(S)<\frac{1}{12}), or S belongs to the family \mathcal{S} of the following groups: \displaystyle\psl(2,7),\psl(2,8),\psl(2,11),\psl(2,13),\psl(3,3), \displaystyle\psl(3,4),\psp(6,2),\psu(4,2),\mathrm{M}_{11},\M_{12}. The GAP Library of Tables of Marks [ 2 ] contains the tables of marks of all almost simple groups whose socle belongs to \mathcal{S} , and using them the exact value of \tilde{\nu}(S) , reported in Table 1 , can be easily obtained (see Section 3 for more details). S \tilde{\nu}(S) \mathrm{PSL}(2,7) \tfrac{3}{56}\approx 0.0536 \mathrm{PSL}(2,8) \tfrac{1}{56}\approx 0.0179 \mathrm{PSL}(2,11) \tfrac{2}{165}\approx 0.0121 \mathrm{PSL}(2,13) \tfrac{3}{364}\approx 0.0082 \mathrm{PSL}(3,3) \tfrac{1}{234}\approx 0.0043 \mathrm{PSL}(3,4) \tfrac{5}{4032}\approx 0.0032 \mathrm{PSU}(4,2) \tfrac{67}{23760}\approx 0.0026 \mathrm{PSp}(6,2) \tfrac{1}{4536}\approx 0.0002 \M_{11} \tfrac{1}{440}\approx 0.0023 \M_{12} \tfrac{7}{11880}\approx 0.0006 Table 1. \tilde{\nu}(S) for S\in\mathcal{S} Now assume that S=\mathrm{Alt}(n) . Using GAP, and in particular the library of table marks, it can be checked that \tilde{\nu}(\mathrm{Alt}(5))=\tfrac{1}{12},\ \tilde{\nu}(\mathrm{Alt}(6))=\tfrac{1}{36},\ \tilde{\nu}(\mathrm{Alt}(7))=\tfrac{1}{210},\ \tilde{\nu}(\mathrm{Alt}(8))=\tfrac{19}{9720},\ \tilde{\nu}(\mathrm{Alt}(9))=\tfrac{1}{2160}. Now suppose n\geq 10 and, for 1\leq i\leq n, let X_{i}=\{s\in\perm(n)\mid s(i)=i\}\cong\perm(n-1),\quad Y_{i}=X_{i}\cap\mathrm{Alt}(n)\cong\mathrm{Alt}(n-1). Consider a_{1},a_{2}\in\perm(n) and let Y_{a_{1},a_{2}}=\langle a_{1},a_{2},\mathrm{Alt}(n)\rangle. Moreover, let \Omega_{0}=\{(s_{1},s_{2})\in(\mathrm{Alt}(n))^{2}\mid\langle s_{1}a_{1},s_{2}a_{2}\rangle=Y_{a_{1},a_{2}}\}, and, for 1\leq i\leq n , let \Omega_{i}=\{(s_{1},s_{2})\in(\mathrm{Alt}(n))^{2}\mid\langle s_{1}a_{1},s_{2}a_{2}\rangle=Y_{a_{1},a_{2}}\cap X_{i}\}. Notice that \Omega_{0},\Omega_{1},\dots,\Omega_{n} are disjoint subsets of (\mathrm{Alt}(n))^{2} and \Omega_{0}\cup\Omega_{1}\cup\dots\cup\Omega_{n}\subseteq(\mathrm{Alt}(n))^{2}\setminus\mathcal{N}_{a_{1},a_{2}}(\perm(n),\mathrm{Alt}(n)). It follows that \nu_{a_{1},a_{2}}(\perm(n),\mathrm{Alt}(n))\leq 1-\sum_{0\leq i\leq n}\frac{|\Omega_{i}|}{|\mathrm{Alt}(n)|^{2}}. For 1\leq i\leq n, there exists s_{1},s_{2}\in\mathrm{Alt}(n) such that \langle s_{1}a_{1},s_{2}a_{2}\rangle\leq X_{i} and |\Omega_{i}|=|Y_{i}|^{2}\pi_{s_{1}a_{1},s_{2}a_{2}}(X_{i},Y_{i})\geq|\mathrm{Alt}(n-1)|^{2}\tilde{\pi}(\mathrm{Alt}(n-1)). Moreover |\Omega_{0}|\geq|\mathrm{Alt}(n)|^{2}\tilde{\pi}(\mathrm{Alt}(n)). Hence \displaystyle\nu_{a_{1},a_{2}}(\perm(n),\mathrm{Alt}(n)) \displaystyle\leq 1-\tilde{\pi}(\mathrm{Alt}(n))-n\left(\frac{|\mathrm{Alt}(n-1)|^{2}\tilde{\pi}(\mathrm{Alt}(n-1))}{|\mathrm{Alt}(n)|^{2}}\right) \displaystyle=1-\tilde{\pi}(\mathrm{Alt}(n))-\frac{\tilde{\pi}(\mathrm{Alt}(n-1))}{n}. It follows from [ 9 ] that if m\geq 9 then \tilde{\pi}(\mathrm{Alt}(m))\geq\tilde{\pi}(\mathrm{Alt}(9))=\frac{15403}{18144} and therefore for, n\geq 10, \displaystyle\tilde{\nu}(\mathrm{Alt}(n)) \displaystyle\leq 1-\tilde{\pi}(\mathrm{Alt}(n))-\frac{\tilde{\pi}(\mathrm{Alt}(n-1))}{n} \displaystyle\leq 1-\frac{15403}{18144}\left(1+\frac{1}{10}\right)=\frac{12007}{181440}\approx 0,0622.\qed Proof of Theorem 1 . Assume that G is a nonsolvable finite group. Then G admits a nonabelian chief factor X/Y . In particular G/C_{G}(X/Y) has a unique minimal normal subgroup, and this minimal normal subgroup is isomorphic to X/Y. It can be easily proved that \nu(G)\leq\nu(G/N) for every normal subgroup N of G (see for example [ 5 , Lemma 2.1] ). This implies that \nu(G)\leq\nu(G/C_{G}(X/Y)). Moreover, by Proposition 4 , \nu(G/C_{G}(X/Y))\leq\tilde{\nu}(S), where S is a composition factor of X/Y. Since, by Proposition 5 , \tilde{\nu}(S)\leq 1/12 , we conclude \nu(G)\leq\nu(G/C_{G}(X/Y))\leq\tilde{\nu}(S)\leq\frac{1}{12}.\qed 3. Details on the computation of \tilde{\nu}(S) To compute the exact value of \tilde{\nu}(S) , with S\in\mathcal{S} , the following observation is useful. Lemma 6 . Let N be a normal subgroup of a finite group G , and let \Omega be the set of pairs (g_{1},g_{2})\in G^{2} such that \langle g_{1},g_{2}\rangle N=G . Then the value of \nu_{g_{1},g_{2}}(G,N) is the same for every choice of (g_{1},g_{2})\in\Omega . Proof. Denote by \mathcal{H} the set of 2-generated subgroups H of G that are nilpotent and such that G=HN . Assume that (g_{1},g_{2})\in\Omega and, for every H\in\mathcal{H} , let \Delta(g_{1},g_{2},H) be the set of (n_{1},n_{2}) such that H=\langle n_{1}g_{1},n_{2}g_{2}\rangle. Then \mathcal{N}_{g_{1},g_{2}}(G,N) is the disjoint union of the subsets \Delta(g_{1},g_{2},H) , with H ranging over \mathcal{H}. Fix H\in\mathcal{H} . Since HN=G, there exist n_{1},n_{2}\in N such that g_{1}=n_{1}h_{1} and g_{2}=n_{2}h_{2} and therefore (n_{1}^{-1},n_{2}^{-1})\in\Delta(g_{1},g_{2},H) . Moreover \Delta(g_{1},g_{2},H)=\{(m_{1},m_{2})\in(N\cap H)^{2}\mid\langle(m_{1}n_{1})(n_{1}^{-1}g_{1}),(m_{2}n_{2})(n_{2}^{-1}g_{2})\rangle=H\}. It follows from the main result in [ 3 ] that |\Delta(g_{1},g_{2},H)|/|N\cap H|^{2} does not depend on the choice of (g_{1},g_{2}) and coincides with the conditional probability that two randomly chosen elements of H generate H given that they generate H modulo H\cap N. Consequently, the cardinality of the sets \Delta(g_{1},g_{2},H) , and hence also that of their disjoint union, does not depend on the choice of (g_{1},g_{2}) . ∎ Given S\in\mathcal{S} , for every almost simple group G with \mathrm{soc}(G)=S , we use the table of marks of G to determine the nilpotent 2-generated subgroups of G that supplement \mathrm{soc}(G) , and for each such subgroup we compute the number of ordered pairs that generate it. Let \tau(G,S) be the sum of the numbers thus obtained, divided by |S|^{2} and by the number of pairs that generate G/S . The resulting number coincides with \nu_{g_{1},g_{2}}(G,S) , where \langle g_{1},g_{2}\rangle S=G . Finally, \tilde{\nu}(S) is the maximum of the values \tau(G,S) as G ranges over the almost simple groups with socle S . The case which requires more attention is when S=\psl(3,4) . In this case, \mathrm{Out}S\cong C_{2}\times\perm(3) and there are 6 different subgroups S\leq G\leq\aut S such that G/S is nilpotent and the values of \tau(G,S) are as in the following table. G S S.{2_{1}} S.{2_{2}} S.{2_{3}} S.2^{2} S.6 \tau(G,S) \tfrac{5}{4032} \tfrac{13}{4032} \tfrac{19}{6720} \tfrac{5}{4032} \tfrac{13}{4032} \tfrac{1}{2520} Table 2. \tau(G,S) when \soc(G)=\psl(3,4) In particular \tilde{\nu}(\psl(3,4))=13/4032.