K^{F} -INVARIANTS IN IRREDUCIBLE REPRESENTATIONS OF G^{F}, WHEN G = GL_{n}

ANTHONY HENDERSON

Abstract. Using a general result of Lusztig, we give explicit formulas for the dimensions of K^{F} -invariants in irreducible representations of G^{F}, when G = GL_{n}, F : G → G is a Frobenius map, and K is an F -stable subgroup of finite index in G^{θ} for some involution θ : G → G commuting with F. The proofs use some combinatorial facts about characters of symmetric groups.

Introduction

Let G be a connected reductive group defined over a finite field 𝔽_{q} of odd characteristic. Let F be the Frobenius morphism on G, whose fixed-point subgroup G^{F} = G(𝔽_{q}) is finite. Let θ : G → G be an involution of algebraic groups commuting with F, and K an F -stable subgroup of the fixed-point subgroup G^{θ} which contains (G^{θ})^{∘}. The homogeneous variety G/K is a symmetric space, and the set of cosets G^{F} /K^{F} might reasonably be called a finite symmetric space.

The classification of irreducible representations of G^{F} (in characteristic 0) was completed by Lusztig in the mid-1980s (see [7] for a general statement, and [9] and [10] for more details). A roughly analogous problem for symmetric spaces is that of determining the dimension of the K^{F} -invariants in each irreducible representation of G^{F}; equivalently, calculating the multiplicities ⟨χ, Ind_{K^{F}}^{G^{F}} (1)⟩ for every irreducible character χ of G^{F}. A large first step towards solving this problem was Lusztig’s calculation, in [11], of ⟨tr(−, R_{T}^{λ}), Ind_{K^{F}}^{G^{F}} (1)⟩ for every Deligne-Lusztig virtual representation R_{T}^{λ} of G^{F}. In [12], Lusztig proceeded to solve the problem completely in the case when G^{F} = (G′)^{F^{2}}, K^{F} = (G′)^{F}, and G′ has connected centre.

This paper is devoted to the solution of this problem when G is a general linear group (with either split or non-split F, so that G^{F} is either GL_{n}(𝔽_{q}) or U_{n}(𝔽_{q^{2}})) and θ is arbitrary. (The solution for the case G^{F} = GL_{n}(𝔽_{q}), K^{F} = Sp_{n}(𝔽_{q}) was found by Bannai, Kawanaka, and Song in [1, §4].) For such G, the functions tr(−, R_{T}^{λ}) form a basis of the class functions, and the transition matrix from this basis to that of the irreducible characters has a simple form. So calculating ⟨χ, Ind_{K^{F}}^{G^{F}} (1)⟩ for χ ∈ Ĝ^{F} from Lusztig’s result is a straightforward (though not entirely easy) matter. Two justifications for presenting it in detail are the potential interest of the answers, and the pleasantness of the symmetric group combinatorics involved.

After recalling Lusztig’s formula and the character theory of the finite general linear and unitary groups in §1, we traverse the various cases in §§2-4, which could be thought of as a theme and variations: the theme, or underlying pattern, is stated in its simplest form in §2.1 (the case already known from [1]), and successive subsections follow the same pattern with progressively more elaborate alterations. The key combinatorial results we need along the way are all proved in §5.

Date: July 24, 2001.

To give some idea of how the formulas in §§2-4 connect with previously known results, we here extract the answers for unipotent irreducible characters. For both GL_{n}(𝔽_{q}) and U_{n}(𝔽_{q^{2}}), these are parametrized by partitions of n, say ρ ↦ χ^{ρ} ∈ Ĝ^{F}. (In our convention χ^{(n)} is the trivial character and χ^{(1^{n})} is the Steinberg character.) Recall that a signed tableau of shape ρ is a signed Young diagram of shape ρ where signs alternate across rows, modulo permutations of rows of equal length. Its signature is the number of boxes of each sign. For all other combinatorial notation, see §1.2. We have:

(n even) ⟨χ^{ρ}, Ind_{Sp_{n}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ = \{ \begin{matrix}1, & if ρ is even \\ 0, & otherwise \end{matrix}

(for the general GL_{n}(𝔽_{q})/Sp_{n}(𝔽_{q}) case, see Theorem 2.1.1);

(n even) ⟨χ^{ρ}, Ind_{Sp_{n}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = \{ \begin{matrix}1, & if ρ is even \\ 0, & otherwise \end{matrix}

(for the general U_{n}(𝔽_{q^{2}})/Sp_{n}(𝔽_{q}) case, see Theorem 2.2.1);

⟨χ^{ρ}, Ind_{GL_{n^{+}} (𝔽_{q})×GL_{n^{−}} (𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ = the number of signed tableaux of shape ρ′ and signature (n^{+}, n^{−})

(for the general GL_{n}(𝔽_{q})/(GL_{n^{+}}(𝔽_{q}) × GL_{n^{−}}(𝔽_{q})) case, see Theorem 3.1.1);

(n even)  ⟨χ^{ρ}, Ind_{GL_{n/2}(𝔽_{q^{2}} )}^{GL_{n}(𝔽_{q})}(1)⟩ = the number of signed tableaux of shape ρ′, stable under changing all signs
= \{ \begin{matrix}1, & if ρ is even \\ 0, & otherwise \end{matrix}

(for the general GL_{n}(𝔽_{q})/GL_{n/2}(𝔽_{q^{2}}) case, see Theorem 3.2.1);

⟨χ^{ρ}, Ind_{U_{n^{+}} (𝔽_{q^{2}} )×U_{n^{−}} (𝔽_{q^{2}} )}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = the number of signed tableaux of shape ρ′ and signature (n^{+}, n^{−}), stable under inverting all rows

(for the general U_{n}(𝔽_{q^{2}})/(U_{n^{+}}(𝔽_{q^{2}}) × U_{n^{−}}(𝔽_{q^{2}})) case, see Theorem 3.3.1);

(n even)  ⟨χ^{ρ}, Ind_{U_{n/2}(𝔽_{q^{4}} )}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = the number of signed tableaux of shape ρ′, stable under changing all signs and inverting all rows

= \{ \begin{matrix}∏_{i}(m_{2i}(ρ′) + 1), & if 2 | m_{2i+1}(ρ′), ∀i \\ 0, & otherwise \end{matrix}

(for the general U_{n}(𝔽_{q^{2}})/U_{n/2}(𝔽_{q^{4}}) case, see Theorem 3.4.1);

(n odd)  ⟨χ^{ρ}, Ind_{O_{n}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ = ⟨χ^{ρ}, Ind_{SO_{n}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩

= the number of signed tableaux of shape ρ, modulo changing all signs

= \frac{1}{2} ∏_{i}(m_{i}(ρ) + 1)

K^{F} -INVARIANTS IN IRREDUCIBLE REPRESENTATIONS OF G^{F}, WHEN G = GL_{n} 3

(for the general GL_{n}(𝔽_{q})/O_{n}(𝔽_{q}) and GL_{n}(𝔽_{q})/SO_{n}(𝔽_{q}) cases, see Theorem 4.1.1);

(n even) ⟨χ^{ρ}, Ind_{O_{n}^{ϵ}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ = \{ \begin{matrix}⌈\frac{1}{2} ∏_{i}(m_{i}(ρ) + 1)⌉, & if ρ′ is even, ϵ = + \\ ⌊\frac{1}{2} ∏_{i}(m_{i}(ρ) + 1)⌋, & if ρ′ is even, ϵ = − \\ \frac{1}{2} ∏_{i}(m_{i}(ρ) + 1), & otherwise, \end{matrix}

⟨χ^{ρ}, Ind_{SO_{n}^{ϵ}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ = \{ \begin{matrix}⌈\frac{1}{2} ∏_{i}(m_{i}(ρ) + 1)⌉ + 1, & if ρ′ is even, ϵ = + \\ ⌊\frac{1}{2} ∏_{i}(m_{i}(ρ) + 1)⌋, & if ρ′ is even, ϵ = − \\ \frac{1}{2} ∏_{i}(m_{i}(ρ) + 1), & otherwise \end{matrix}

(for the general GL_{n}(𝔽_{q})/O_{n}^{±}(𝔽_{q}) and GL_{n}(𝔽_{q})/SO_{n}^{±}(𝔽_{q}) cases, see Theorem 4.2.1);

(n odd)  ⟨χ^{ρ}, Ind_{O_{n}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = ⟨χ^{ρ}, Ind_{SO_{n}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩

= the number of signed tableaux of shape ρ, stable under inverting all rows, modulo changing all signs

= \{ \begin{matrix}\frac{1}{2} ∏_{i}(m_{2i+1}(ρ) + 1), & if 2 | m_{2i}(ρ), ∀i \\ 0, & otherwise \end{matrix}

(for the general U_{n}(𝔽_{q^{2}})/O_{n}(𝔽_{q}) and U_{n}(𝔽_{q^{2}})/SO_{n}(𝔽_{q}) cases, see Theorem 4.3.1);

(n even) ⟨χ^{ρ}, Ind_{O_{n}^{ϵ}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = \{ \begin{matrix}⌈\frac{1}{2} ∏_{i}(m_{2i}(ρ) + 1)⌉, & if ρ′ is even, ϵ = + \\ ⌊\frac{1}{2} ∏_{i}(m_{2i}(ρ) + 1)⌋, & if ρ′ is even, ϵ = − \\ \frac{1}{2} ∏_{i}(m_{2i}(ρ) + 1), & if 2 | m_{2i+1}(ρ), ∀i, but ρ′ not even \\ 0, & otherwise \end{matrix}

⟨χ^{ρ}, Ind_{SO_{n}^{ϵ}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = \{ {⌈\frac{1}{2} ∏_{i} \atop (m_{2i+1}(ρ) + 1)⌉+⌈\frac{1}{2} ∏_{i}(m_{2i}(ρ) + 1)⌉,} if ρ′ is even, ϵ = + {⌊\frac{1}{2} ∏_{i} \atop (m_{2i+1}(ρ) + 1)⌋+⌊\frac{1}{2} ∏_{i}(m_{2i}(ρ) + 1)⌋,} if ρ′ is even, ϵ = − \begin{aligned}\frac{1}{2} & ∏_{i} \\ & (m_{2i+1}(ρ) + 1), \end{aligned} if 2 | m_{2i}(ρ), ∀i, but ρ′ not even \begin{aligned}\frac{1}{2} & ∏_{i} \\ & (m_{2i}(ρ) + 1), \end{aligned} if 2 | m_{2i+1}(ρ), ∀i, but ρ′ not even 0, otherwise

(for the general U_{n}(𝔽_{q^{2}})/O_{n}^{±}(𝔽_{q}) and U_{n}(𝔽_{q^{2}})/SO_{n}^{±}(𝔽_{q}) cases, see Theorem 4.4.1).

Readers experienced in the representation theory of real Lie groups will find these answers familiar. When G^{F} ≅ GL_{n}(𝔽_{q}) and K is split over 𝔽_{q}, the connection is as follows. The unipotent irreducible characters are the constituents of Ind_{B^{F}}^{G^{F}} (1) where B is an F -stable Borel subgroup, and the Hecke algebra H(G^{F} , B^{F}) is the specialization at q of the abstract Hecke algebra of S_{n}. This means that the above multiplicities ⟨χ^{ρ}, Ind_{K^{F}}^{G^{F}} (1)⟩ can be interpreted as the multiplicity of the irreducible representation of S_{n} labelled by ρ in the coherent continuation representation associated to the appropriate real form of GL_{n}(ℂ) and the trivial infinitesimal character. For example, our answer in the case of GL_{n}(𝔽_{q})/(GL_{n^{+}}(𝔽_{q}) × GL_{n^{−}}(𝔽_{q})) could be deduced from the results in [2], and the fact that

⟨χ^{ρ}, Ind_{O_{n}^{(+)}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ = ∑_{\begin{matrix}n^{+}+n^{−}=n \\ n^{+}≥n^{−} \end{matrix}}⟨χ^{ρ′}, Ind_{GL_{n^{+}} (𝔽_{q})×GL_{n^{−}} (𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩

is, in view of the results of [15], a manifestation of Vogan duality. Similarly, our results below for non-unipotent irreducible characters connect with representations of non-trivial infinitesimal character.

However, when K is not split over 𝔽_{q}, or when G^{F} ≅ U_{n}(𝔽_{q^{2}}) (in which case the Hecke algebra H(G^{F} , B^{F}) is a specialization with unequal parameters of the abstract Hecke algebra of type B_{⌊ \frac{n}{2} ⌋}, see [8, §10]), the significance of our results for real groups is unclear. In particular, I do not know an a priori explanation for the fact that

⟨χ^{ρ}, Ind_{O_{n}^{(+)}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = ∑_{\begin{matrix}n^{+}+n^{−}=n \\ n^{+}≥n^{−} \end{matrix}}⟨χ^{ρ′}, Ind_{U_{n^{+}} (𝔽_{q^{2}} )×U_{n^{−}} (𝔽_{q^{2}} )}^{U_{n}(𝔽_{q^{2}} )}(1)⟩.

Part of this work was done at the Massachusetts Institute of Technology. I would like to thank George Lusztig for all his help and encouragement, and Peter Trapa and David Vogan for stimulating discussions.

Contents

Introduction 1 1. Review of Known Results 5 1.1. Lusztig’s Formula 5 1.2. Combinatorial Notation 6 1.3. Character Theory of GL_{n}(𝔽_{q}) and U_{n}(𝔽_{q^{2}}) 7 2. Cases where G/K = GL_{n}/Sp_{n} 9 2.1. The GL_{n}(𝔽_{q})/Sp_{n}(𝔽_{q}) Case 10 2.2. The U_{n}(𝔽_{q^{2}})/Sp_{n}(𝔽_{q}) Case 12 3. Cases where G/K = GL_{n}/(GL_{n^{+}} × GL_{n^{−}}) 14 3.1. The GL_{n}(𝔽_{q})/(GL_{n^{+}}(𝔽_{q}) × GL_{n^{−}}(𝔽_{q})) Case 14 3.2. The GL_{n}(𝔽_{q})/GL_{n/2}(𝔽_{q^{2}}) Case 18 3.3. The U_{n}(𝔽_{q^{2}})/(U_{n^{+}}(𝔽_{q^{2}}) × U_{n^{−}}(𝔽_{q^{2}})) Case 19 3.4. The U_{n}(𝔽_{q^{2}})/U_{n/2}(𝔽_{q^{4}}) Case 23 4. Cases where G/K = GL_{n}/O_{n} or GL_{n}/SO_{n} 25 4.1. The GL_{n}(𝔽_{q})/O_{n}(𝔽_{q}) and GL_{n}(𝔽_{q})/SO_{n}(𝔽_{q}) Cases (n odd) 25 4.2. The GL_{n}(𝔽_{q})/O_{n}^{±}(𝔽_{q}) and GL_{n}(𝔽_{q})/SO_{n}^{±}(𝔽_{q}) Cases (n even) 29 4.3. The U_{n}(𝔽_{q^{2}})/O_{n}(𝔽_{q}) and U_{n}(𝔽_{q^{2}})/SO_{n}(𝔽_{q}) Cases (n odd) 33 4.4. The U_{n}(𝔽_{q^{2}})/O_{n}^{±}(𝔽_{q}) and U_{n}(𝔽_{q^{2}})/SO_{n}^{±}(𝔽_{q}) Cases (n even) 38 5. Combinatorics of the Symmetric Group 42 References 49

1. Review of Known Results

In this section, we recall the theorem of Lusztig which underpins all our results, and the relevant parts of the character theory of the finite general linear and unitary groups in a convenient form. In between, we introduce some combinatorial notation to be used throughout the paper.

1.1. Lusztig’s Formula. Let k be the algebraic closure of a finite field 𝔽_{q} of odd cardinality q. Let G be a connected reductive group over k defined over 𝔽_{q}, with Frobenius map F : G → G. Let θ : G → G be an involution of algebraic groups commuting with F, and K an F -stable subgroup of the fixed-point subgroup G^{θ} which contains (G^{θ})^{∘}. Fix a prime l not dividing q. All representations and characters of finite groups in this paper will be over \overline{ℚ_{l}}.

A pair (T, λ) means an F -stable maximal torus T and a character λ : T^{F} → \overline{ℚ_{l}}^{×}. We have a conjugation action of G^{F} on the set of pairs. In [3], Deligne and Lusztig attached to each pair a virtual representation R_{T}^{λ} of G^{F}, depending only on the G^{F} -orbit of (T, λ). (In general, “most” of the R_{T}^{λ} are irreducible up to sign, and every irreducible representation occurs in some R_{T}^{λ}; when G = GL_{n} the situation is even better, as we will see below.) The main result of [11] is a formula for

\frac{1}{|K^{F} |} ∑_{k∈K^{F}} tr(k, R_{T}^{λ}) = ⟨tr(−, R_{T}^{λ}), Ind_{K^{F}}^{G^{F}} (1)⟩.

After some trivial adjustments, it reads as follows. Define

Θ_{T} = {f ∈ G | θ(f^{−1}T f) = f^{−1}T f}.

Then T acts on Θ_{T} by left multiplication and K acts by right multiplication. If B is a Borel subgroup containing T, the obvious map T\Θ_{T} /K → B\G/K is a bijection (see [11, Proposition 1.3]), so T \Θ_{T} /K is in bijection with the set of K-orbits on the flag variety. For any f ∈ Θ_{T}^{F}, define ϵ_{T,f} : (T ∩ fKf^{−1})^{F} → {±1} by

ϵ_{T,f}(t) = (−1)^{𝔽_{q}-rank(Z_{G}((T ∩fKf^{−1})^{∘}))+𝔽_{q}-rank(Z_{G}^{∘}(t)∩Z_{G}((T ∩fKf^{−1})^{∘}))}.

It follows from [11, Proposition 2.3] that ϵ_{T,f} is a group homomorphism which factors through (T ∩ fKf^{−1})^{F} /((T ∩ fKf^{−1})^{∘})^{F}. Note that (T ∩ fKf^{−1})^{∘} = (T ∩ fK^{∘}f^{−1})^{∘}, so that ϵ_{T,f} defined for K ≠ G^{θ} is simply the restriction of ϵ_{T,f} defined for K = G^{θ}. Finally, define

Θ_{T,K,λ}^{F} = {f ∈ Θ_{T}^{F} | λ|_{(T ∩fKf^{−1})^{F}} = ϵ_{T,f}},

a union of T^{F} –K^{F} double cosets. When K is determined by θ, we write Θ_{T,λ}^{F}.

Theorem 1.1.1. (Lusztig, [11, Theorem 3.3])

⟨tr(−, R_{T}^{λ}), Ind_{K^{F}}^{G^{F}} (1)⟩ = ∑_{f∈T^{F} \Θ_{T,K,λ}^{F}/K^{F}} (−1)^{𝔽_{q}-rank(T)+𝔽_{q}-rank(Z_{G}((T ∩fKf^{−1})^{∘}))}.

1.2. Combinatorial Notation. In general, our combinatorial notation always follows [14]. For instance, λ ⊢ n means that λ is a partition of n. The size of a partition λ is written |λ| and its length ℓ(λ); it has parts λ_{1}, λ_{2}, …, λ_{ℓ(λ)}. The transpose partition is λ′. We define

n(λ) = ∑_{i}(i − 1)λ_{i} = ∑_{i} \binom{λ′_{i}}{2}.

The multiplicity of i as a part of λ is written m_{i}(λ). We say that λ is even if all its parts are even, or equivalently if 2|m_{i}(λ′), ∀i.

By abuse of notation, we will write simply ν for an (unspecified) element of the symmetric group S_{|ν|} of cycle-type ν. The sign of such an element is written ϵ_{ν}, and the size of its centralizer Z_{S_{|ν|}}(ν) is z_{ν}. We will have much to do with involutions in Z_{S_{|ν|}}(ν). Note that if w is such an involution, then the cycles of the element ν (i.e. the parts of the partition ν) are of three types:

(1) those fixed pointwise by w;

(2) those fixed by w, not pointwise – these must be of even length, and w must “turn the cycle half-way around”; and

(3) those moved (i.e. not fixed) by w.

If w is fixed-point free, all cycles of ν fixed by w must be of type (2). We label the irreducible characters of S_{|ν|} as {χ^{λ} | λ ⊢ |ν|} as in [14, I.7], and write χ_{ν}^{λ} for the value of χ^{λ} at ν, so that χ^{(|ν|)} is the trivial character and χ_{ν}^{λ′} = ϵ_{ν}χ_{ν}^{λ}.

We define the signature of an involution in S_{n} with signed fixed points to be (n^{+}, n^{−}), where n^{+} + n^{−} = n and

n^{+} − n^{−} = |{fixed points signed +}| − |{fixed points signed −}|.

In this context and that of signed tableaux (for which see the introduction), “signature (p^{+}, p^{−}) + ℤ” means “signature (p^{+} + s, p^{−} + s) for some s ∈ ℤ”.

As in [14, Chapter IV], we will need to consider the system of maps 𝔽̂_{q^{e}}^{×} → 𝔽̂_{q^{e′}}^{×}

for e | e′ (the transpose of the norm map), and its limit L = colim 𝔽̂_{q^{e}}^{×}. Let σ denote

the q-th power map on both k^{×} and L, so that (k^{×})^{σ^{e}} ≅ 𝔽_{q^{e}}^{×}, L^{σ^{e}} ≅ 𝔽̂_{q^{e}}^{×} for all e ≥ 1. Write ⟨·, ·⟩^{σ^{e}} : (k^{×})^{σ^{e}} × L^{σ^{e}} → \overline{ℚ_{l}}^{×} for the canonical pairing. Let ι denote the inverse map on k^{×} and L, and write σ̃ for ισ, the (−q)-th power map. (Note that σ̃^{2} = σ^{2}.) We also have a canonical pairing ⟨·, ·⟩^{σ̃^{e}} : (k^{×})^{σ̃^{e}} × L^{σ̃^{e}} → \overline{ℚ_{l}}^{×} (the same as ⟨·, ·⟩^{σ^{e}} if e is even).

We will fix some set of representatives for the orbits of the group ⟨σ⟩ generated by σ on L, and call it ⟨σ⟩\L. Similarly define ⟨σ̃⟩\L. For ξ ∈ ⟨σ⟩\L, let m_{ξ} = |⟨σ⟩.ξ|, in other words the smallest e ≥ 1 such that σ^{e}(ξ) = ξ. Let d_{ξ} = ⟨−1, ξ⟩^{σ^{m_{ξ}}}, which equals 1 if L^{σ^{m_{ξ}}} contains square roots of ξ, and −1 if it does not. Fix once and for all an element ζ of L^{σ} such that d_{ζ} = −1. Then for any ξ ∈ L,

m_{ζξ} = m_{ξ}, and d_{ζξ} = (−1)^{m_{ξ}}d_{ξ}.

Similarly define m̃_{ξ}, d̃_{ξ} and ζ̃, using σ̃ instead of σ, so that for any ξ ∈ L,

m̃_{ζ̃ξ} = m̃_{ξ}, and d̃_{ζ̃ξ} = (−1)m̃_{ξ} d̃_{ξ}.

1.3. Character Theory of GL_{n}(𝔽_{q}) and U_{n}(𝔽_{q^{2}}). For the remainder of the paper, we specialize the context of §1.1 drastically, to the case when G ≅ GL_{n}, for some positive integer n. More concretely, let V be a vector space over k of dimension n and let G = GL(V). There are two kinds of 𝔽_{q}-structures on G, split and non-split. A Frobenius map F : G → G is split if it is induced by some Frobenius map F_{V} : V → V, in the sense that

F_{V} (gv) = F (g)F_{V} (v), ∀g ∈ G, v ∈ V.

Then G^{F} ≅ Aut_{𝔽_{q}}(V^{F_{V}}) ≅ GL_{n}(𝔽_{q}), the finite general linear group. If F is a nonsplit Frobenius map, there exists some outer involution θ′ of G commuting with F, and for any such θ′, θ′F is a split Frobenius map. In this case G^{F} ≅ U_{n}(𝔽_{q^{2}}), the finite unitary group.

First consider the case when F : G → G is a split Frobenius map. Let P_{n} be the set of collections of partitions \underline{λ} = (λ_{α})_{α∈k^{×}}, almost all zero, such that ∑_{α∈k^{×}} |λ_{α}| = n. Let P_{n}^{σ} be the subset of P_{n} consisting of all \underline{λ} such that λ_{σ(α)} = λ_{α} for all α. It is well known that there is a natural bijection between P_{n}^{σ} and the set of conjugacy classes in G^{F}. Dually, let P̂_{n} be the set of collections of partitions \underline{ν} = (ν_{ξ})_{ξ∈L}, almost all zero, such that ∑_{ξ∈L} |ν_{ξ}| = n. Let P̂_{n}^{σ} be the subset of P̂_{n} of all \underline{ν} such that ν_{σ(ξ)} = ν_{ξ} for all ξ. Note that for \underline{ν} ∈ P̂_{n}^{σ},

∑_{ξ∈⟨σ⟩\L} m_{ξ}|ν_{ξ}| = n.

We can define a bijection between P̂_{n}^{σ} and the set of G^{F} -orbits of pairs (T, λ) as above, so that if (T, λ) is in the orbit corresponding to \underline{ν}:

(1) the eigenlines of T can be labelled

{L_{(ξ,j,i)} | ξ ∈ ⟨σ⟩\L, 1 ≤ j ≤ ℓ(ν_{ξ}), i ∈ ℤ/m_{ξ}(ν_{ξ})_{j}ℤ}

so that under the resulting isomorphism

T ≅ ∏_{ξ∈⟨σ⟩\L} ∏_{j=1}^{ℓ(ν_{ξ})} \underbrace{k^{×} × ⋯ × k^{×}}_{m_{ξ}(ν_{ξ})_{j} factors},

F on T corresponds to cyclic permutation of each group of factors k^{×}, composed with σ;

(2) consequently,

T^{F} ≅ ∏_{ξ∈⟨σ⟩\L} ∏_{j=1}^{ℓ(ν_{ξ})}(k^{×})^{σ^{m_{ξ}(ν_{ξ})_{j}}};

(3) under this isomorphism, λ corresponds to

∏_{ξ∈⟨σ⟩\L} ∏_{j=1}^{ℓ(ν_{ξ})}⟨·, ξ⟩^{σ^{m_{ξ}(ν_{ξ})_{j}}} .

For \underline{ν}, \underline{ρ} ∈ P̂_{n}^{σ}, we write |\underline{ν}| = |\underline{ρ}| to mean that |ν_{ξ}| = |ρ_{ξ}| for all ξ.

For \underline{ν} ∈ P̂_{n}^{σ}, let B\underline{ν} = tr(−, R_{T}^{λ}) for (T, λ) in the corresponding G^{F} -orbit. As proved by Lusztig in [6], these coincide with the basic characters defined by Green in [4]. (In particular, their values are computable, but this is not relevant here.) Green’s main result on the character theory of GL_{n}(𝔽_{q}) states that for any \underline{ρ} ∈ P̂_{n}^{σ},

χ\underline{ρ} := (−1)^{n+∑_{ξ∈⟨σ⟩\L} |ρ_{ξ}|} ∑_{\begin{matrix}\underline{ν}∈P̂_{n}^{σ} \\ |\underline{ν}|=|\underline{ρ}| \end{matrix}} ( ∏_{ξ∈⟨σ⟩\L}(z_{ν_{ξ}})^{−1}χ_{ν_{ξ}}^{ρ_{ξ}} ) B\underline{ν}

is an irreducible character of G^{F}, and all irreducible characters arise in this way for unique \underline{ρ} ∈ P̂_{n}^{σ}. (See also [14, Chapter IV] and [1, Theorem 1.2.10]. Note that Macdonald’s parameters in P̂_{n}^{σ} differ from those of [1] by transposing all partitions; we are following the convention of [1].) In words, the transition matrix between the basic characters and the irreducible characters is formed from the character table of various symmetric groups. Inverting this matrix, we have that for any \underline{ν} ∈ P̂_{n}^{σ},

(1.3.1)  B\underline{ν} = (−1)^{n+∑_{ξ∈⟨σ⟩\L} |ν_{ξ}|} ∑_{\begin{matrix}\underline{ρ}∈P̂_{n}^{σ} \\ |\underline{ρ}|=|\underline{ν}| \end{matrix}} ( ∏_{ξ∈⟨σ⟩\L} χ_{ν_{ξ}}^{ρ_{ξ}} ) χ\underline{ρ}.

There is an obvious action of L^{σ} on P̂_{n}^{σ}, and in particular, for \underline{ν} ∈ P̂_{n}^{σ}, ζ.\underline{ν} is well defined. For any η ∈ L^{σ}, B_{η.\underline{ν}} and χ^{η.\underline{ρ}} are the result of multiplying B\underline{ν} and χ\underline{ρ} by the one-dimensional character ⟨det(·), η⟩^{σ} of G^{F}. The unipotent irreducible characters referred to in the introduction are those χ\underline{ρ} for which ρ_{ξ} = 0 unless ξ = 1. (In the introduction we parametrized these by ρ = ρ_{1}.)

The case when F : G → G is a non-split Frobenius map is less well known, but very similar, in fact mostly identical once σ is replaced by σ̃, m_{ξ} by m̃_{ξ}, and so on. Define P̂σ̃ ; n in the obvious way. Again, for any \underline{ν} ∈ P̂σ̃ ; n,

∑_{ξ∈⟨σ̃⟩\L} m̃_{ξ}|ν_{ξ}| = n.

We can define a bijection between P̂σ̃ ; n and the set of G^{F} -orbits of pairs (T, λ) as above, so that if (T, λ) is in the orbit corresponding to \underline{ν}:

(1) the eigenlines of T can be labelled

{L_{(ξ,j,i)} | ξ ∈ ⟨σ̃⟩\L, 1 ≤ j ≤ ℓ(ν_{ξ}), i ∈ ℤ/m̃_{ξ}(ν_{ξ})_{j}ℤ}

so that under the resulting isomorphism

T ≅ ∏_{ξ∈⟨σ̃⟩\L} ∏_{j=1}^{ℓ(ν_{ξ})} \underbrace{k^{×} × ⋯ × k^{×}}_{m̃_{ξ}(ν_{ξ})_{j} factors},

F on T corresponds to cyclic permutation of each group of factors k^{×}, composed with σ̃;

(2) consequently,

T^{F} ≅ ∏_{ξ∈⟨σ̃⟩\L} ∏_{j=1}^{ℓ(ν_{ξ})}(k^{×})^{σ̃^{m̃_{ξ}(ν_{ξ})_{j}}};

(3) under this isomorphism, λ corresponds to

∏_{ξ∈⟨σ̃⟩\L} ∏_{j=1}^{ℓ(ν_{ξ})}⟨·, ξ⟩^{σ̃^{m̃_{ξ}(ν_{ξ})_{j}}} .

For \underline{ν}, \underline{ρ} ∈ P̂σ̃ ; n, we write |\underline{ν}| = |\underline{ρ}| to mean that |ν_{ξ}| = |ρ_{ξ}| for all ξ.

For \underline{ν} ∈ P̂σ̃ ; n, let B\underline{ν} = tr(−, R_{T}^{λ}) for (T, λ) in the corresponding G^{F} -orbit. The extension of Green’s result to the non-split case was proved by Lusztig and Srinivasan in [13, Theorem 3.2]: in our notation, for any \underline{ρ} ∈ P̂_{n}^{σ},

χ\underline{ρ} := (−1)^{⌈ \frac{n}{2} ⌉+∑_{ξ∈⟨σ̃⟩\L} m̃_{ξ}n(ρ′_{ξ})+|ρ_{ξ}|} ∑_{\begin{matrix}\underline{ν}∈P̂σ̃ ; n \\ |\underline{ν}|=|\underline{ρ}| \end{matrix}} ( ∏_{ξ∈⟨σ̃⟩\L}(z_{ν_{ξ}})^{−1}χ_{ν_{ξ}}^{ρ_{ξ}} ) B\underline{ν}

is an irreducible character of G^{F}, and all irreducible characters arise in this way for unique \underline{ρ} ∈ P̂σ̃ ; n. Inverting, we see that for any \underline{ν} ∈ P̂σ̃ ; n,

(1.3.2)  B\underline{ν} = ∑_{\begin{matrix}\underline{ρ}∈P̂σ̃ ; n \\ |\underline{ρ}|=|\underline{ν}| \end{matrix}}(−1)^{⌈ \frac{n}{2} ⌉+∑_{ξ∈⟨σ̃⟩\L} m̃_{ξ}n(ρ′_{ξ})+|ρ_{ξ}|} ( ∏_{ξ∈⟨σ̃⟩\L} χ_{ν_{ξ}}^{ρ_{ξ}} ) χ\underline{ρ}.

Again, the obvious action of Lσ̃ on P̂σ̃ ; n corresponds to multiplication by onedimensional characters, and the unipotent irreducible characters of the introduction are those χ\underline{ρ} for which ρ_{ξ} = 0 unless ξ = 1. (In contrast to the case of GL_{n}(𝔽_{q}), not all the unipotent characters are constituents of Ind_{B^{F}}^{G^{F}} (1) for an F -stable Borel subgroup B.)

2. Cases where G/K = GL_{n}/Sp_{n}

In this section, we suppose that V has a nondegenerate symplectic form ⟨·, ·⟩ (so in particular n is even), and that θ : G → G is the involution defined by

⟨θ(g)v, v′⟩ = ⟨v, g^{−1}v′⟩, ∀g ∈ G, v, v′ ∈ V.

Since G^{θ} = Sp(V, ⟨·, ·⟩) is connected, K must be equal to it. So G/K is the symmetric space GL_{n}/Sp_{n}.

Let T be any maximal torus of G. Let L_{1}, ⋯, L_{n} be some ordering of the eigenlines of T, so that T ≅ GL(L_{1}) × ⋯ × GL(L_{n}). This ordering gives an identification of W (T) with S_{n}. If f ∈ Θ_{T}, then f^{−1}T f is a θ-stable maximal torus, so for a unique w_{f} ∈ W (T), we have

(f^{−1}L_{i})^{⊥} = ⨁_{i′≠w_{f} (i)} f^{−1}L_{i′}, for all i,

and

θ|_{f^{−1}T f} = Ad(f^{−1}) ∘ w_{f} ∘ Ad(f) ∘ ι.

Since θ is an involution, w_{f} must be also; and since ⟨·, ·⟩ is a symplectic form, every line is orthogonal to itself, so that w_{f} is fixed-point free. Note that

T ∩ fKf^{−1} = {t ∈ T | t|_{L_{w_{f} (i)}} = (t|_{L_{i}})^{−1}, ∀i}

is connected, and Z_{G}(T ∩ fKf^{−1}) = T.

Clearly the association f ↦ w_{f} induces a map

T \Θ_{T} /K → {fixed-point free involutions in W (T)}.

Proposition 2.0.1. This map is a bijection.

Proof. This is very well known, especially when translated into the language of flags via the connection mentioned in §1.1. Surjectivity can be proved by an explicit construction, and injectivity is easy by induction. □

2.1. The GL_{n}(𝔽_{q})/Sp_{n}(𝔽_{q}) Case. In this subsection, let F : G → G be a split Frobenius map which commutes with θ. So F is induced by a Frobenius map F_{V} on V which respects ⟨·, ·⟩. One has G^{F} ≅ GL_{n}(𝔽_{q}), K^{F} ≅ Sp_{n}(𝔽_{q}). The following result was obtained by a different (and simpler) method in [1, §4]:

Theorem 2.1.1. For any \underline{ρ} ∈ P̂_{n}^{σ},

⟨χ\underline{ρ}, Ind_{Sp_{n}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ = \{ \begin{matrix}1, & if all ρ_{ξ} are even \\ 0, & otherwise. \end{matrix}

By the results in §1.3, it is equivalent to prove that for any \underline{ν} ∈ P̂_{n}^{σ},

(2.1.1)  ⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ = ∏_{ξ∈⟨σ⟩\L} ∑_{\begin{matrix}ρ_{ξ}⊢|ν_{ξ}| \\ ρ_{ξ} even \end{matrix}} χ_{ν_{ξ}}^{ρ_{ξ}}.

Note that the sign in (1.3.1) disappears since n is even, and |ρ| is even for even ρ.

We aim to deduce this from Lusztig’s general formula for the left-hand side (Theorem 1.1.1 above). This entails successively analysing the components of the formula in our combinatorial terms, until we are reduced to a fact about class functions on the symmetric group, whose proof we defer to §5. This basic strategy will be repeated in every case; the main reason for including the present case, where the result is not new, is that it serves as the prototype for the following subsections.

Let T be an F -stable maximal torus, and λ : T^{F} → \overline{ℚ_{l}}^{×} a character, for which (T, λ) is in the G^{F} -orbit corresponding to \underline{ν} ∈ P̂_{n}^{σ}. Label the eigenlines of T as

{L_{(ξ,j,i)} | ξ ∈ ⟨σ⟩\L, 1 ≤ j ≤ ℓ(ν_{ξ}), i ∈ ℤ/m_{ξ}(ν_{ξ})_{j}ℤ}

as in §1.3. Thus W (T) is identified with the set of permutations of these triples (ξ, j, i). Let w_{F} ∈ W (T) be the permutation induced by F_{V}, i.e. the disjoint product of the cycles (ξ, j, i) ↦ (ξ, j, i + 1).

Lemma 2.1.2. For f ∈ Θ_{T}, w_{F (f)} = w_{F} w_{f}w_{F}^{−1} ∈ W (T).

Proof. Applying F_{V} to the equation

(f^{−1}L_{(ξ,j,i)})^{⊥} = ⨁_{(ξ′,j′,i′)≠w_{f} (ξ,j,i)} f^{−1}L_{(ξ′,j′,i′)},

we find that

(F (f)^{−1}L_{w_{F} (ξ,j,i)})^{⊥} = ⨁_{(ξ′,j′,i′)≠w_{f} (ξ,j,i)} F (f)^{−1}L_{w_{F} (ξ′,j′,i′)}.

The Lemma follows. □

Corollary 2.1.3. The map f ↦ w_{f} induces a bijection

T^{F} \Θ_{T}^{F} /K^{F} →^{∼} {fixed-point free involutions in Z_{W (T)}(w_{F})}.

Proof. One need only note that since T, K, and all T ∩ fKf^{−1} are connected, T^{F} \Θ_{T}^{F} /K^{F} = (T \Θ_{T} /K)^{F}. □

Now any fixed-point free involution w in Z_{W (T)}(w_{F}) is of the form

(ξ, j, i) ↦ (w̄(ξ, j), i + i(w, ξ, j))

where w̄ is an involution of the set of (ξ, j) preserving the quantity m_{ξ}(ν_{ξ})_{j}, and i(w, ξ, j) ∈ ℤ/m_{ξ}(ν_{ξ})_{j}ℤ satisfies i(w, w̄(ξ, j)) = −i(w, ξ, j) and i(w, ξ, j) = \frac{1}{2}m_{ξ}(ν_{ξ})_{j} if w̄(ξ, j) = (ξ, j). Let Z′ be the set of fixed-point free involutions w in Z_{W (T)}(w_{F}) for which w̄(ξ, j) = (ξ, j′) (for some j′) and m_{ξ} | i(w, ξ, j) hold for all (ξ, j).

Lemma 2.1.4. For f ∈ Θ_{T}^{F}, f ∈ Θ_{T,λ}^{F} ⇔ w_{f} ∈ Z′.

Proof. Since T ∩ fKf^{−1} is connected, ϵ_{T,f} = 1, so

Θ_{T,λ}^{F} = {f ∈ Θ_{T}^{F} | λ|_{(T ∩fKf^{−1})^{F}} = 1}.

Now

(T ∩ fKf^{−1})^{F} = {t ∈ T^{F} | t|_{L_{w_{f} (ξ,j,i)}} = (t|_{L_{(ξ,j,i)}})^{−1}}

≅ {(α_{(ξ,j)}) ∈ ∏_{(ξ,j)}(k^{×})^{σ^{m_{ξ}(ν_{ξ})_{j}}} |σ^{i(w_{f} ,ξ,j)}(α_{w̄_{f} (ξ,j)}) = α_{(ξ,j)}^{−1}, ∀ξ, j }

= \{(β_{(ξ,j)}) ∈ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \end{matrix}}(k^{×})^{ισ^{\frac{1}{2} m_{ξ}(ν_{ξ})_{j}}} \}

× \{(γ_{(ξ,j)}) ∈ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)≠(ξ,j) \end{matrix}}(k^{×})^{σ^{m_{ξ}(ν_{ξ})_{j}}} |γ_{w̄_{f} (ξ,j)} = σ^{−i(w_{f} ,ξ,j)}(γ_{(ξ,j)}^{−1})\}

and in this form λ|_{(T ∩fKf^{−1})^{F}} is the function

((β_{(ξ,j)}), (γ_{(ξ,j)})) ↦ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \end{matrix}}⟨β_{(ξ,j)}, ξ⟩^{σ^{m_{ξ}(ν_{ξ})_{j}}} ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)≠(ξ,j) \end{matrix}}⟨γ_{(ξ,j)}, ξ⟩^{σ^{m_{ξ}(ν_{ξ})_{j}}} .

Hence λ|_{(T ∩fKf^{−1})^{F}} = 1 if and only if:

• whenever w̄_{f}(ξ, j) = (ξ, j), we have σ^{\frac{1}{2} m_{ξ}(ν_{ξ})_{j}}(ξ) = ξ; and
• whenever w̄_{f}(ξ, j) = (ξ′, j′) ≠ (ξ, j), we have ξσ^{−i(w_{f} ,ξ,j)}(ξ′)^{−1} = 1.

By definition of ⟨σ⟩\L and m_{ξ}, this is equivalent to w_{f} ∈ Z′. □

Corollary 2.1.5. The map f ↦ w_{f} induces a bijection T^{F} \Θ_{T,λ}^{F}/K^{F} →^{∼} Z′.

As noted above, Z_{G}((T ∩ fKf^{−1})^{∘}) = T, so Lusztig’s formula becomes

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ = |T^{F} \Θ_{T,λ}^{F}/K^{F} | = |Z′|.

For any w ∈ Z′, we define a permutation ŵ of the set of triples (ξ, j, s) with ξ ∈ ⟨σ⟩\L, 1 ≤ j ≤ ℓ(ν_{ξ}), and s ∈ ℤ/(ν_{ξ})_{j}ℤ by

ŵ(ξ, j, s) = (w̄(ξ, j), s + \frac{i(w, ξ, j)}{m_{ξ}}).

It is clear that w ↦ ŵ defines a bijection between Z′ and the set of fixed-point free involutions in ∏_{ξ∈⟨σ⟩\L} Z_{S_{|ν_{ξ}|}}(ν_{ξ}). Now we apply the combinatorial fact (for which see §5):

(2.1.2)  |\{fixed-point free involutions in Z_{S_{|ν|}}(ν)\}| = ∑_{\begin{matrix}λ⊢|ν| \\ λ even \end{matrix}} χ_{ν}^{λ}.

This gives Equation (2.1.1) and hence Theorem 2.1.1.

Remark 2.1.6. For the only time in this paper, let us consider what happens if G = SL_{n} instead of GL_{n}. As is well known, every irreducible character of SL_{n}(𝔽_{q}) occurs with multiplicity one in the restriction of a unique irreducible character of GL_{n}(𝔽_{q}). From Theorem 2.1.1 we see that if not all ρ_{ξ} are even, the constituents of Res_{SL_{n}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(χ\underline{ρ}) are all orthogonal to Ind_{Sp_{n}(𝔽_{q})}^{SL_{n}(𝔽_{q})}(1); and if all ρ_{ξ} are even, all but

one of these constituents are orthogonal to Ind_{Sp_{n}(𝔽_{q})}^{SL_{n}(𝔽_{q})}(1), and one, say χ^{\underline{ρ},⋆}, has

⟨χ^{\underline{ρ},⋆}, Ind_{Sp_{n}(𝔽_{q})}^{SL_{n}(𝔽_{q})}(1)⟩ = 1.

The question remains to identify χ^{\underline{ρ},⋆} in terms of a parametrization of the constituents of Res_{SL_{n}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(χ\underline{ρ}), say the parametrization given by [10, Proposition 5.1]. It seems likely that the answer will be such that the almost characters of SL_{n}(𝔽_{q}) have inner product with Ind_{Sp_{n}(𝔽_{q})}^{SL_{n}(𝔽_{q})}(1) which is an algebraic integer. Indeed, this may be a general property of finite reductive symmetric spaces G^{F} /K^{F} in the setting of §1.1. (It holds when G^{F} = (G′)^{F} × (G′)^{F}, K^{F} = (G′)^{F} by the orthonormality of almost characters; it also holds when G^{F} = (G′)^{F^{2}}, K^{F} = (G′)^{F} by [12, Corollary 1.6].)

2.2. The U_{n}(𝔽_{q^{2}})/Sp_{n}(𝔽_{q}) Case. Now we keep the assumptions from before §2.1, but take F to be instead a non-split Frobenius map which commutes with θ. So θF is a split Frobenius map as above, induced by a Frobenius map F_{V} on V which respects ⟨·, ·⟩. One has G^{F} ≅ U_{n}(𝔽_{q^{2}}), K^{F} ≅ Sp_{n}(𝔽_{q}). In this case the result is:

Theorem 2.2.1. For any \underline{ρ} ∈ P̂σ̃ ; n,

⟨χ\underline{ρ}, Ind_{Sp_{n}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = \{ \begin{matrix}1, & if all ρ_{ξ} are even \\ 0, & otherwise. \end{matrix}

We will prove the equivalent statement that for any \underline{ν} ∈ P̂σ̃ ; n,

(2.2.1)  ⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ = ∏_{ξ∈⟨σ̃⟩\L} ∑_{\begin{matrix}ρ_{ξ}⊢|ν_{ξ}| \\ ρ_{ξ} even \end{matrix}} χ_{ν_{ξ}}^{ρ_{ξ}}.

Note that the sign in (1.3.2) disappears because n(ρ′) ≡ \frac{|ρ|}{2} mod 2 for any even ρ, so that

∑_{ξ∈⟨σ̃⟩\L} m̃_{ξ}n(ρ′_{ξ}) ≡ ∑_{ξ∈⟨σ̃⟩\L} m̃_{ξ} \frac{|ρ_{ξ}|}{2} ≡ \frac{n}{2} mod 2.

Let T be an F -stable maximal torus, and λ : T^{F} → \overline{ℚ}^{×} a character, for which (T, λ) is in the G^{F} -orbit corresponding to \underline{ν} ∈ P̂σ̃ ; n. Label the eigenlines of T as

{L_{(ξ,j,i)} | ξ ∈ ⟨σ̃⟩\L, 1 ≤ j ≤ ℓ(ν_{ξ}), i ∈ ℤ/m̃_{ξ}(ν_{ξ})_{j}ℤ}

as in §1.3. Thus W (T) is identified with the set of permutations of these triples (ξ, j, i). Let w_{F} ∈ W (T) be the disjoint product of the cycles (ξ, j, i) ↦ (ξ, j, i + 1). It is no longer simply the permutation induced by F_{V}; instead

F_{V} (L_{(ξ,j,i)}) = ⋂_{(ξ′,j′,i′)≠w_{F} (ξ,j,i)} L_{(ξ′,j′,i′)}^{⊥}.

Lemma 2.2.2. For f ∈ Θ_{T}, w_{F (f)} = w_{F} w_{f}w_{F}^{−1} ∈ W (T).

Proof. Applying F_{V} to the equation

(f^{−1}L_{(ξ,j,i)})^{⊥} = ⨁_{(ξ′,j′,i′)≠w_{f} (ξ,j,i)} f^{−1}L_{(ξ′,j′,i′)},

we find that

\begin{aligned}(θ(F (f))^{−1} ⋂_{\begin{matrix}(ξ′′,j′′,i′′) \\ ≠w_{F} (ξ,j,i) \end{matrix}} L_{(ξ′′,j′′,i′′)}^{⊥})^{⊥} & = ⨁_{\begin{matrix}(ξ′,j′,i′) \\ ≠w_{f} (ξ,j,i) \end{matrix}} θ(F (f))^{−1} ⋂_{\begin{matrix}(ξ′′′,j′′′,i′′′) \\ ≠w_{F} (ξ′,j′,i′) \end{matrix}} L_{(ξ′′′,j′′′,i′′′)}^{⊥} \\ & = θ(F (f))^{−1}L_{w_{F} w_{f} (ξ,j,i)}^{⊥}, \end{aligned}

in other words that

F (f)^{−1} ⨁_{(ξ′′,j′′,i′′)≠w_{F} (ξ,j,i)} L_{(ξ′′,j′′,i′′)} = (F (f)^{−1}L_{w_{F} w_{f} (ξ,j,i)})^{⊥} .

The Lemma follows. □

Corollary 2.2.3. The map f ↦ w_{f} induces a bijection

T^{F} \Θ_{T}^{F} /K^{F} →^{∼} {fixed-point free involutions in Z_{W (T)}(w_{F})}.

Proof. This is deduced exactly as in the previous subsection. □

Define Z′ ⊂ Z_{W (T)}(w_{F}) in the same way as before, but with σ̃ instead of σ and m̃_{ξ} instead of m_{ξ}.

Lemma 2.2.4. For f ∈ Θ_{T}^{F}, f ∈ Θ_{T,λ}^{F} ⇔ w_{f} ∈ Z′.

Proof. The proof is exactly analogous to that of Lemma 2.1.4. □

Reasoning exactly as in the previous subsection (with σ̃ instead of σ and m̃_{ξ} instead of m_{ξ}) we get Equation (2.2.1) and hence Theorem 2.2.1.

Remark 2.2.5. One can also give a proof of Theorem 2.2.1 along the lines of the proof of Theorem 2.1.1 in [1, §4]. Indeed, all the results of [1] have analogues for the symmetric space U_{n}(𝔽_{q^{2}})/Sp_{n}(𝔽_{q}), as is explained in [5].

3. Cases where G/K = GL_{n}/(GL_{n^{+}} × GL_{n^{−}})

In this section, we suppose that θ is an inner involution, namely conjugation with respect to s ∈ G such that s^{2} = 1. Let V^{+} be the (+1)-eigenspace and V^{−} the (−1)-eigenspace of s on V, so that V = V^{+}⊕V^{−}. Let n^{+} = dim V^{+}, n^{−} = dim V^{−}, so that n = n^{+} + n^{−}. Since G^{θ} = GL(V^{+}) × GL(V^{−}) is connected, K must be equal to it. So G/K is the symmetric space GL_{n}/(GL_{n^{+}} × GL_{n^{−}}).

Let T be any maximal torus of G. Let L_{1}, ⋯, L_{n} be some ordering of the eigenlines of T. If f ∈ Θ_{T}, then f^{−1}T f is a θ-stable maximal torus, so there is some w_{f} ∈ W (T) = S_{n} such that

s(f^{−1}L_{i}) = f^{−1}L_{w_{f} (i)}

and

θ|_{f^{−1}T f} = Ad(f^{−1}) ∘ w_{f} ∘ Ad(f).

Since θ is an involution, w_{f} must be also. Moreover, if w_{f}(i) = i, then f^{−1}L_{i} is stable under s, so must lie in either V^{+} or V^{−}. Let

ϵ_{f} : {fixed points of w_{f}} → {+, −}

be the resulting association of a sign to each fixed point. Note that

T ∩ fKf^{−1} = {t ∈ T | t|_{L_{w_{f} (i)}} = t|_{L_{i}}, ∀i}

is connected, and

Z_{G}(T ∩ fKf^{−1}) = ∏_{w_{f} (i)=i} GL(L_{i}) × ∏_{{w_{f} (i)≠i}} GL(L_{i} ⊕ L_{w_{f} (i)}).

Clearly the association f ↦ (w_{f}, ϵ_{f}) induces a map

T \Θ_{T} /K → {involutions in W (T) with signed fixed points, signature (n^{+}, n^{−})}.

Proposition 3.0.1. This map is a bijection.

Proof. As with Proposition 2.0.1, this is well known when translated in terms of K-orbits on the flag variety, and easy to prove (see for instance [16]). □

3.1. The GL_{n}(𝔽_{q})/(GL_{n^{+}}(𝔽_{q})×GL_{n^{−}}(𝔽_{q})) Case. In this subsection, assume that F : G → G is a split Frobenius map such that F (s) = s. So F is induced by a Frobenius map F_{V} on V which stabilizes V^{+} and V^{−}. One has G^{F} ≅ GL_{n}(𝔽_{q}), K^{F} ≅ GL_{n^{+}}(𝔽_{q}) × GL_{n^{−}}(𝔽_{q}). The result is:

Theorem 3.1.1. For any \underline{ρ} ∈ P̂_{n}^{σ},

⟨χ\underline{ρ}, Ind_{GL_{n^{+}} (𝔽_{q})×GL_{n^{−}} (𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ = \{ \begin{matrix}a(n^{+}, n^{−}, ρ′_{1}), & if ρ_{ξ} = ρ_{ξ^{−1}}, ∀ξ and ρ′_{−1} is even \\ 0, & otherwise, \end{matrix}

where a(p^{+}, p^{−}, λ′) denotes the number (possibly zero) of signed tableaux of shape λ′ and signature (p^{+}, p^{−}) + ℤ.

By (1.3.1), it is equivalent to say that for any \underline{ν} ∈ P̂_{n}^{σ},

\begin{aligned}⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ & = ( ∑_{ρ_{1}⊢|ν_{1}|} a(n^{+}, n^{−}, ρ′_{1})χ_{ν_{1}}^{ρ_{1}})( ∑_{\begin{matrix}ρ_{−1}⊢|ν_{−1}| \\ ρ′_{−1} even \end{matrix}} χ_{ν_{−1}}^{ρ_{−1}}) \\ & × ∏_{\begin{matrix}ξ∈⟨σ⟩\L \\ ξ^{−1}∈⟨σ⟩.ξ \\ ξ≠1,−1 \end{matrix}}((−1)^{|ν_{ξ}|} ∑_{ρ_{ξ}⊢|ν_{ξ}|} χ_{ν_{ξ}}^{ρ_{ξ}}) ∏_{\begin{matrix}\{ξ_{1}≠ξ_{2}\}∈⟨σ⟩\L \\ ξ_{1}^{−1}∈⟨σ⟩.ξ_{2} \end{matrix}} δ_{ν_{ξ_{1}} ν_{ξ_{2}}} z_{ν_{ξ_{1}}} . \end{aligned} (3.1.1)

Here the sign (−1)^{n+∑_{ξ∈⟨σ⟩\L} |ν_{ξ}|} = (−1)^{∑_{ξ∈⟨σ⟩\L}(m_{ξ}+1)|ν_{ξ}|} in (1.3.1) is simplified by noting that m_{1} = m_{−1} = 1, m_{ξ} is even for ξ ≠ 1, −1 such that ξ^{−1} ∈ ⟨σ⟩.ξ, and ν_{ξ_{1}} and ν_{ξ_{2}} must be equal whenever ξ_{1}^{−1} ∈ ⟨σ⟩.ξ_{2}, ξ_{1} ≠ ξ_{2} in order for the right-hand side to be nonzero.

Let (T, λ) be a pair in the G^{F} -orbit corresponding to \underline{ν} ∈ P̂_{n}^{σ}. Define \{L_{(ξ,j,i)}\} and w_{F} exactly as in §2.1.

Lemma 3.1.2. For f ∈ Θ_{T}, w_{F (f)} = w_{F} w_{f}w_{F}^{−1}, and ϵ_{F (f)} = ϵ_{f} ∘ w_{F}^{−1}.

Proof. Applying F_{V} to the equation

s(f^{−1}L_{(ξ,j,i)}) = f^{−1}L_{w_{f} (ξ,j,i)},

we find that

s(F (f)^{−1}L_{w_{F} (ξ,j,i)}) = F (f)^{−1}L_{w_{F} w_{f} (ξ,j,i)},

whence the first statement. If f^{−1}L_{(ξ,j,i)} lies in V^{+} or V^{−}, then F_{V} (f^{−1}L_{(ξ,j,i)}) = F (f)^{−1}L_{w_{F} (ξ,j,i)} lies in the same one, whence the second statement. □

Corollary 3.1.3. The map f ↦ (w_{f}, ϵ_{f}) induces a bijection

T^{F} \Θ_{T}^{F} /K^{F} →^{∼} \{ \begin{matrix}involutions in Z_{W (T)}(w_{F}) with signed fixed points, \\ signs stable under w_{F}, of signature (n^{+}, n^{−}) \end{matrix} \} .

Proof. Again, this follows directly from Proposition 3.0.1 and the Lemma, since T, K, and all T ∩ fKf^{−1} are connected. □

Given an involution w in Z_{W (T)}(w_{F}), we can define as before an involution w̄ of the set of (ξ, j) preserving m_{ξ}(ν_{ξ})_{j}, and the quantities i(w, ξ, j) ∈ ℤ/m_{ξ}(ν_{ξ})_{j}ℤ satisfying i(w, w̄(ξ, j)) = −i(w, ξ, j). (Since w is not assumed to be fixed-point free, it is possible that w̄(ξ, j) = (ξ, j) and i(w, ξ, j) = 0.) For ξ ∈ ⟨σ⟩\L, let ξ^{∨} be the chosen representative in the ⟨σ⟩-orbit of ξ^{−1}. Define i_{0}(ξ) ∈ ℤ/m_{ξ}ℤ by

ξ^{∨} = σ^{i_{0}(ξ)}(ξ^{−1}).

Clearly i_{0}(ξ^{∨}) = −i_{0}(ξ). Let Z′ be the set of involutions w in Z_{W (T)}(w_{F}) such that w̄(ξ, j) = (ξ^{∨}, j′) (for some j′) and i(w, ξ, j) ≡ i_{0}(ξ) mod m_{ξ} hold for all (ξ, j), and w̄(−1, j) = (−1, j) implies i(w, −1, j) = \frac{1}{2}(ν_{−1})_{j}.

Lemma 3.1.4. For f ∈ Θ_{T}^{F}, f ∈ Θ_{T,λ}^{F} ⇔ w_{f} ∈ Z′.

Proof. Since T ∩ fKf^{−1} is connected, ϵ_{T,f} = 1, so

Θ_{T,λ}^{F} = {f ∈ Θ_{T}^{F} | λ|_{(T ∩fKf^{−1})^{F}} = 1}.

Now

(T ∩ fKf^{−1})^{F} ≅ {(α_{(ξ,j)}) ∈ ∏_{(ξ,j)}(k^{×})^{σ^{m_{ξ}(ν_{ξ})_{j}}} |σ^{i(w_{f} ,ξ,j)}(α_{w̄_{f} (ξ,j)}) = α_{(ξ,j)}, ∀ξ, j }

= \{(α_{(ξ,j)}) ∈ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}}(k^{×})^{σ^{m_{ξ}(ν_{ξ})_{j}}} \}

× \{(β_{(ξ,j)}) ∈ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)= \frac{1}{2} m_{ξ}(ν_{ξ})_{j} \end{matrix}}(k^{×})^{σ^{\frac{1}{2} m_{ξ}(ν_{ξ})_{j}}} \}

× \{(γ_{(ξ,j)}) ∈ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)≠(ξ,j) \end{matrix}}(k^{×})^{σ^{m_{ξ}(ν_{ξ})_{j}}} |γ_{w̄_{f} (ξ,j)} = σ^{−i(w_{f} ,ξ,j)}(γ_{(ξ,j)})\}

and in this form λ|_{(T ∩fKf^{−1})^{F}} is the function

((α_{(ξ,j)}), (β_{(ξ,j)}), (γ_{(ξ,j)})) ↦ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}}⟨α_{(ξ,j)}, ξ⟩^{σ^{m_{ξ}(ν_{ξ})_{j}}}

× ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)= \frac{1}{2} m_{ξ}(ν_{ξ})_{j} \end{matrix}}⟨β_{(ξ,j)}, ξ⟩^{σ^{m_{ξ}(ν_{ξ})_{j}}}

× ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)≠(ξ,j) \end{matrix}}⟨γ_{(ξ,j)}, ξ⟩^{σ^{m_{ξ}(ν_{ξ})_{j}}} .

Hence λ|_{(T ∩fKf^{−1})^{F}} = 1 if and only if:

• whenever w̄_{f}(ξ, j) = (ξ, j) and i(w_{f}, ξ, j) = 0, we have ξ = 1;

• whenever w̄_{f}(ξ, j) = (ξ, j) and i(w_{f}, ξ, j) = \frac{1}{2}m_{ξ}(ν_{ξ})_{j}, σ^{\frac{1}{2} m_{ξ}(ν_{ξ})_{j}}(ξ) = ξ^{−1}; and

• whenever w̄_{f}(ξ, j) = (ξ′, j′) ≠ (ξ, j), ξσ^{−i(w_{f} ,ξ,j)}(ξ′) = 1.

By definition of m_{ξ}, ξ^{∨}, and i_{0}(ξ), this is equivalent to w_{f} ∈ Z′. □

Corollary 3.1.5. The map f ↦ (w_{f}, ϵ_{f}) induces a bijection

T^{F} \Θ_{T,λ}^{F}/K^{F} →^{∼} \{ \begin{matrix}w ∈ Z′ with signed fixed points, \\ signs stable under w_{F}, signature (n^{+}, n^{−}) \end{matrix} \} .

Now the 𝔽_{q}-rank of T is ∑_{ξ∈⟨σ⟩\L} ℓ(ν_{ξ}) = |{(ξ, j)}|, and that of

Z_{G}((T ∩ fKf^{−1})^{∘}) = ∏_{\begin{matrix}(ξ,j,i) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}} GL(L_{(ξ,j,i)})

× ∏_{\begin{matrix}\{(ξ,j,i),(ξ,j,i+ \frac{1}{2} m_{ξ}(ν_{ξ})_{j})\} \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)= \frac{1}{2} m_{ξ}(ν_{ξ})_{j} \end{matrix}} GL(L_{(ξ,j,i)} ⊕ L_{(ξ,j,i+ \frac{1}{2} m_{ξ}(ν_{ξ})_{j})})

× ∏_{\begin{matrix}\{(ξ,j,i),(ξ′,j′,i′)\} \\ w̄_{f} (ξ,j)=(ξ′,j′)≠(ξ,j) \\ i′=i+i(w_{f} ,ξ,j) \end{matrix}} GL(L_{(ξ,j,i)} ⊕ L_{(ξ′,j′,i′)})

is

|\{(ξ, j)\}| + |\{(ξ, j) | w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = \frac{1}{2}m_{ξ}(ν_{ξ})_{j}\}|.

So Lusztig’s formula becomes

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ = ∑_{f∈T^{F} \Θ_{T,λ}^{F}/K^{F}} (−1)^{|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)= \frac{1}{2} m_{ξ}(ν_{ξ})_{j}\}|}

= ∑_{w ∈ Z′ with signed fixed points, signs stable under w_{F}, signature (n^{+}, n^{−})} (−1)^{|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)= \frac{1}{2} m_{ξ}(ν_{ξ})_{j}\}|}.

Suppose w ∈ Z′. Note that if ξ^{−1} ∈ ⟨σ⟩.ξ, ξ ≠ 1, −1, we have ξ^{∨} = ξ and i_{0}(ξ) = \frac{m_{ξ}}{2} . If w̄(ξ, j) = (ξ, j) for such ξ, we have both i(w, ξ, j) ≡ \frac{m_{ξ}}{2} mod m_{ξ} and i(w, ξ, j) = −i(w, ξ, j), whence i(w, ξ, j) = \frac{1}{2}m_{ξ}(ν_{ξ})_{j} and (ν_{ξ})_{j} is odd. We define a permutation ŵ of the set of triples (ξ, j, s) (as in §2.1) by

ŵ(ξ, j, s) = (w̄(ξ, j), s + \frac{i(w, ξ, j) − i_{0}(ξ, j)}{m_{ξ}}),

where we choose some preimages i_{0}(ξ, j) ∈ ℤ/m_{ξ}(ν_{ξ})_{j}ℤ of i_{0}(ξ) ∈ ℤ/m_{ξ}ℤ, equal to 0 if ξ = 1 or −1, equal to \frac{1}{2}m_{ξ}(ν_{ξ})_{j} if ξ^{−1} ∈ ⟨σ⟩.ξ, ξ ≠ 1, −1, w̄(ξ, j) = (ξ, j), and otherwise arbitrarily subject to the constraint i_{0}(w̄(ξ, j)) = −i_{0}(ξ, j). It is clear that w ↦ ŵ defines a bijection between Z′ and the set of involutions ŵ of the set of triples (ξ, j, s), commuting with the disjoint product of the cycles (ξ, j, s) ↦ (ξ, j, s + 1), such that ŵ(ξ, j, s) = (ξ^{∨}, j′, s′) (for some j′, s′) holds always, ŵ has no fixed points of the form (−1, j, s), and ξ^{−1} ∈ ⟨σ⟩.ξ, ξ ≠ 1, −1, ŵ(ξ, j, s) = (ξ, j, s′) implies that (ν_{ξ})_{j} is odd. Hence

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ = ∑_{involutions w_{1} ∈ Z_{S_{|ν_{1}|}} (ν_{1}) with signed fixed points, signs stable under ν_{1}, signature (n^{+}, n^{−}) + ℤ} (−1)^{|\{cycles of ν_{1} fixed by w_{1}, not pointwise\}|}

× ∑_{fixed-point free involutions w_{−1} ∈ Z_{S_{|ν_{−1}|}} (ν_{−1})} (−1)^{|{cycles of ν_{−1} fixed by w_{−1}}|}

× ∏_{\begin{matrix}ξ∈⟨σ⟩\L \\ ξ^{−1}∈⟨σ⟩.ξ \\ ξ≠1,−1 \end{matrix}} ( ∑_{involutions w_{ξ} ∈ Z_{S_{|ν_{ξ}|}} (ν_{ξ}) fixing no even cycles of ν_{ξ}} (−1)^{|\{cycles of ν_{ξ} fixed by w_{ξ}\}|})

× ∏_{\begin{matrix}\{ξ_{1}≠ξ_{2}\}∈⟨σ⟩\L \\ ξ_{1}^{−1}∈⟨σ⟩.ξ_{2} \end{matrix}} |\{ \begin{matrix}bijections \{1, ⋯, |ν_{ξ_{1}}|\} → \{1, ⋯, |ν_{ξ_{2}}|\} \\ intertwining ν_{ξ_{1}} and ν_{ξ_{2}} \end{matrix} \}| .

Now we apply the following combinatorial facts (see §5):

∑_{involutions w ∈ Z_{S_{|ν|}} (ν) with signed fixed points, signs stable under ν, signature (p^{+}, p^{−}) + ℤ} (−1)^{|{cycles of ν fixed by w, not pointwise}|} = ∑_{λ⊢|ν|} a(p^{+}, p^{−}, λ′)χ_{ν}^{λ}, (3.1.2)

(3.1.3)  ∑_{fixed-point free involutions w ∈ Z_{S_{|ν|}} (ν)} (−1)^{|\{cycles of ν fixed by w\}|} = ∑_{\begin{matrix}λ⊢|ν| \\ λ′ even \end{matrix}} χ_{ν}^{λ}, and

(3.1.4)  ∑_{involutions w ∈ Z_{S_{|ν|}} (ν) fixing no even cycles of ν} (−1)^{|{cycles of ν fixed by w}|} = (−1)^{|ν|} ∑_{λ⊢|ν|} χ_{ν}^{λ}.

These give Equation (3.1.1) and hence Theorem 3.1.1.

3.2. The GL_{n}(𝔽_{q})/GL_{n/2}(𝔽_{q^{2}}) Case. Now keep the assumptions from before §3.1, but take F to be a split Frobenius map such that F (s) = −s. (Then F still commutes with θ.) So F is induced by a Frobenius map F_{V} on V which interchanges V^{+} and V^{−}, whence n is even and n^{+} = n^{−} = \frac{n}{2}. One has G^{F} ≅ GL_{n}(𝔽_{q}), K^{F} ≅ GL_{n/2}(𝔽_{q^{2}}). The result is:

Theorem 3.2.1. For any \underline{ρ} ∈ P̂_{n}^{σ},

⟨χ\underline{ρ}, Ind_{GL_{n/2}(𝔽_{q^{2}} )}^{GL_{n}(𝔽_{q})}(1)⟩ = \{ \begin{matrix}1, & if ρ_{ξ} = ρ_{ξ^{−1}}, ∀ξ, ρ_{1} is even, and ρ′_{−1} is even \\ 0, & otherwise. \end{matrix}

By (1.3.1), it is equivalent to say that for any \underline{ν} ∈ P̂_{n}^{σ},

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ = ( ∑_{\begin{matrix}ρ_{1}⊢|ν_{1}| \\ ρ_{1} even \end{matrix}} χ_{ν_{1}}^{ρ_{1}})( ∑_{\begin{matrix}ρ_{−1}⊢|ν_{−1}| \\ ρ′_{−1} even \end{matrix}} χ_{ν_{−1}}^{ρ_{−1}})

× ∏_{\begin{matrix}ξ∈⟨σ⟩\L \\ ξ^{−1}∈⟨σ⟩.ξ \\ ξ≠1,−1 \end{matrix}}((−1)^{|ν_{ξ}|} ∑_{ρ_{ξ}⊢|ν_{ξ}|} χ_{ν_{ξ}}^{ρ_{ξ}}) ∏_{\begin{matrix}\{ξ_{1}≠ξ_{2}\}∈⟨σ⟩\L \\ ξ_{1}^{−1}∈⟨σ⟩.ξ_{2} \end{matrix}} δ_{ν_{ξ_{1}} ν_{ξ_{2}}} z_{ν_{ξ_{1}}} .

(3.2.1)

(For the signs here, see the comments after Equation (3.1.1).)

Let (T, λ) be a pair in the G^{F} -orbit corresponding to \underline{ν} ∈ P̂_{n}^{σ}. Define \{L_{(ξ,j,i)}\} and w_{F} exactly as in §2.1.

Lemma 3.2.2. For f ∈ Θ_{T}, w_{F (f)} = w_{F} w_{f}w_{F}^{−1}, and ϵ_{F (f)} = −ϵ_{f} ∘ w_{F}^{−1}.

Proof. This is an obvious modification of Lemma 3.1.2. □

Corollary 3.2.3. The map f ↦ (w_{f}, ϵ_{f}) induces a bijection

T^{F} \Θ_{T}^{F} /K^{F} →^{∼} \{ \begin{matrix}involutions in Z_{W (T)}(w_{F}) with signed fixed points, \\ signs reversed by w_{F} \end{matrix} \} .

Now Lemma 3.1.4 holds again here, with the same Z′ and exactly the same proof. So arguing as in §3.1, we get

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ = ∑_{w ∈ Z′ with signed fixed points, signs reversed by w_{F}} (−1)^{|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)= \frac{1}{2} m_{ξ}(ν_{ξ})_{j}\}|}.

The rest of the proof is also the same as in §3.1, except that Equation (3.1.2) is replaced by:

(3.2.2)  ∑_{involutions w ∈ Z_{S_{|ν|}} (ν) with signed fixed points, signs reversed by ν} (−1)^{|\{cycles of ν fixed by w, not pointwise\}|} = ∑_{\begin{matrix}λ⊢|ν| \\ λ even \end{matrix}} χ_{ν}^{λ}.

This too will be proved in §5.

3.3. The U_{n}(𝔽_{q^{2}})/(U_{n^{+}}(𝔽_{q^{2}}) × U_{n^{−}}(𝔽_{q^{2}})) Case. Still under the general assumptions of this section, let F : G → G be a non-split Frobenius map for which F (s) = s. Replacing s by a G^{F} -conjugate if necessary, we may assume that there is some nondegenerate symmetric form ⟨·, ·⟩ on V, for which V^{+} and V^{−} are orthogonal, and so that the associated outer involution θ′ : G → G commutes with F. Then θ′F is the split Frobenius map induced by some F_{V} : V → V which respects ⟨·, ·⟩ and fixes V^{+} and V^{−}. One has G^{F} ≅ U_{n}(𝔽_{q^{2}}), K^{F} ≅ U_{n^{+}}(𝔽_{q^{2}}) × U_{n^{−}}(𝔽_{q^{2}}).

Theorem 3.3.1. For any \underline{ρ} ∈ P̂σ̃ ; n,

⟨χ\underline{ρ}, Ind_{U_{n^{+}} (𝔽_{q^{2}} )×U_{n^{−}} (𝔽_{q^{2}} )}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = \{ \begin{matrix}ā(n^{+}, n^{−}, ρ′_{1}), & if ρ_{ξ} = ρ_{ξ^{−1}}, ∀ξ and ρ′_{−1} is even \\ 0, & otherwise, \end{matrix}

where ā(p^{+}, p^{−}, λ′) denotes the number (possibly zero) of signed tableaux of shape λ′ and signature (p^{+}, p^{−}) + ℤ which are fixed under inverting all rows.

By (1.3.2), it is equivalent to say that for any \underline{ν} ∈ P̂σ̃ ; n,

\begin{aligned}⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ & = ( ∑_{ρ_{1}⊢|ν_{1}|}(−1)^{n(ρ_{1})}ā(n^{+}, n^{−}, ρ′_{1})χ_{ν_{1}}^{ρ_{1}})((−1)^{\frac{|ν_{−1}|}{2}} ∑_{\begin{matrix}ρ_{−1}⊢|ν_{−1}| \\ ρ′_{−1} even \end{matrix}} χ_{ν_{−1}}^{ρ_{−1}}) \\ & × ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ ξ^{−1}∈⟨σ̃⟩.ξ \\ ξ≠1,−1 \\ 4|m̃_{ξ} \end{matrix}}((−1)^{|ν_{ξ}|} ∑_{ρ_{ξ}⊢|ν_{ξ}|} χ_{ν_{ξ}}^{ρ_{ξ}}) ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ ξ^{−1}∈⟨σ̃⟩.ξ \\ ξ≠1,−1 \\ 4∤m̃_{ξ} \end{matrix}}( ∑_{ρ_{ξ}⊢|ν_{ξ}|} χ_{ν_{ξ}}^{ρ_{ξ}}) \\ & × ∏_{\begin{matrix}\{ξ_{1}≠ξ_{2}\}∈⟨σ̃⟩\L \\ ξ_{1}^{−1}∈⟨σ̃⟩.ξ_{2} \\ 2|m̃_{ξ_{1}} \end{matrix}} δ_{ν_{ξ_{1}} ν_{ξ_{2}}} z_{ν_{ξ_{1}}} ∏_{\begin{matrix}\{ξ_{1}≠ξ_{2}\}∈⟨σ̃⟩\L \\ ξ_{1}^{−1}∈⟨σ̃⟩.ξ_{2} \\ 2∤m̃_{ξ_{1}} \end{matrix}}(−1)^{|ν_{ξ_{1}} |}δ_{ν_{ξ_{1}} ν_{ξ_{2}}} z_{ν_{ξ_{1}}} . \end{aligned} (3.3.1)

Here the sign (−1)^{⌈ \frac{n}{2} ⌉+∑_{ξ∈⟨σ̃⟩\L} m̃_{ξ}n(ρ′_{ξ})+|ρ_{ξ}|} of (1.3.2) is simplified as follows. If ρ_{ξ} = ρ_{ξ^{−1}} for all ξ and ρ′_{−1} is even, then

n = ∑_{ξ∈⟨σ̃⟩\L} m̃_{ξ}|ρ_{ξ}| ≡ |ρ_{1}| mod 2,

and the sign can be replaced by

\begin{aligned}(−1)^{⌊ \frac{|ρ_{1}|}{2} ⌋+n(ρ′_{1})} & (−1)^{\frac{|ρ_{−1}|}{2} +n(ρ′_{−1})} ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ ξ^{−1}∈⟨σ̃⟩.ξ \\ ξ≠1,−1 \end{matrix}}(−1)^{\frac{m̃_{ξ}}{2} (|ρ_{ξ}|+2n(ρ′_{ξ}))+|ρ_{ξ}|} \\ & × ∏_{\begin{matrix}\{ξ_{1}≠ξ_{2}\}∈⟨σ̃⟩\L \\ ξ_{1}^{−1}∈⟨σ̃⟩.ξ_{2} \end{matrix}}(−1)^{m̃_{ξ_{1}} (|ρ_{ξ_{1}} |+2n(ρ′_{ξ_{1}} ))+2|ρ_{ξ_{1}} |}. \end{aligned}

Then we observe that if ρ′_{−1} is even, n(ρ′_{−1}) is even; and if all even parts of ρ′_{1} occur with even multiplicity (which is necessary for ā(n^{+}, n^{−}, ρ′_{1}) ≠ 0), then the Young diagram of ρ′_{1} (excluding the top left corner if |ρ_{1}| is odd) can be tiled by 2 × 1 dominoes, from which we see that ⌊\frac{|ρ_{1}|}{2}⌋ + n(ρ′_{1}) + n(ρ_{1}) is even.

Let (T, λ) be a pair in the G^{F} -orbit corresponding to \underline{ν} ∈ P̂σ̃ ; n. Define \{L_{(ξ,j,i)}\} and w_{F} exactly as in §2.2.

Lemma 3.3.2. For f ∈ Θ_{T}, w_{F (f)} = w_{F} w_{f}w_{F}^{−1}, and ϵ_{F (f)} = ϵ_{f} ∘ w_{F}^{−1}.

Proof. Applying F_{V} to the equation s(f^{−1}L_{(ξ,j,i)}) = f^{−1}L_{w_{f} (ξ,j,i)}, we get

s (θ′(F (f))^{−1} ⋂_{\begin{matrix}(ξ′,j′,i′) \\ ≠w_{F} (ξ,j,i) \end{matrix}} L_{(ξ′,j′,i′)}^{⊥}) = θ′(F (f))^{−1} ⋂_{\begin{matrix}(ξ′′,j′′,i′′) \\ ≠w_{F} w_{f} (ξ,j,i) \end{matrix}} L_{(ξ′′,j′′,i′′)}^{⊥}.

Taking orthogonals with respect to ⟨·, ·⟩, this becomes

s (F (f)^{−1} ⨁_{\begin{matrix}(ξ′,j′,i′) \\ ≠w_{F} (ξ,j,i) \end{matrix}} L_{(ξ′,j′,i′)}) = F (f)^{−1} ⨁_{\begin{matrix}(ξ′′,j′′,i′′) \\ ≠w_{F} w_{f} (ξ,j,i) \end{matrix}} L_{(ξ′′,j′′,i′′)},

in other words

⨁_{\begin{matrix}(ξ′,j′,i′) \\ ≠w_{F} (ξ,j,i) \end{matrix}} F (f)^{−1}L_{w_{F (f)}(ξ′,j′,i′)} = ⨁_{\begin{matrix}(ξ′′,j′′,i′′) \\ ≠w_{F} w_{f} (ξ,j,i) \end{matrix}} F (f)^{−1}L_{(ξ′′,j′′,i′′)}.

The first statement follows. If, say, f^{−1}L_{(ξ,j,i)} ⊆ V^{+}, then applying F_{V} and taking orthogonals, we find that

F (f)^{−1} ⨁_{\begin{matrix}(ξ′,j′,i′) \\ ≠w_{F} (ξ,j,i) \end{matrix}} L_{(ξ′,j′,i′)} ⊇ V^{−},

whence F (f)^{−1}L_{w_{F} (ξ,j,i)} ⊆ V^{+}. The second statement follows. □

Corollary 3.3.3. The map f ↦ (w_{f}, ϵ_{f}) induces a bijection

T^{F} \Θ_{T}^{F} /K^{F} →^{∼} \{ \begin{matrix}involutions in Z_{W (T)}(w_{F}) with signed fixed points, \\ signs stable under w_{F}, of signature (n^{+}, n^{−}) \end{matrix} \} .

Define Z′ ⊂ Z_{W (T)}(w_{F}) in the same way as in §3.1, but with σ̃ instead of σ and m̃_{ξ} instead of m_{ξ}.

Lemma 3.3.4. For f ∈ Θ_{T}^{F}, f ∈ Θ_{T,λ}^{F} ⇔ w_{f} ∈ Z′.

Proof. The proof is exactly analogous to that of Lemma 3.1.4. □

Corollary 3.3.5. The map f ↦ (w_{f}, ϵ_{f}) induces a bijection

T^{F} \Θ_{T,λ}^{F}/K^{F} →^{∼} \{ \begin{matrix}w ∈ Z′ with signed fixed points, \\ signs stable under w_{F}, signature (n^{+}, n^{−}) \end{matrix} \} .

The important point of difference from §3.1 is the 𝔽_{q}-ranks involved. The 𝔽_{q}- rank of T is now |{(ξ, j) | 2|m̃_{ξ}(ν_{ξ})_{j}}|, and that of

Z_{G}((T ∩ fKf^{−1})^{∘}) = ∏_{\begin{matrix}(ξ,j,i) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}} GL(L_{(ξ,j,i)})

× ∏_{\begin{matrix}\{(ξ,j,i),(ξ,j,i+ \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j})\} \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j} \end{matrix}} GL(L_{(ξ,j,i)} ⊕ L_{(ξ,j,i+ \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j})})

× ∏_{\begin{matrix}\{(ξ,j,i),(ξ′,j′,i′)\} \\ w̄_{f} (ξ,j)=(ξ′,j′)≠(ξ,j) \\ i′=i+i(w_{f} ,ξ,j) \end{matrix}} GL(L_{(ξ,j,i)} ⊕ L_{(ξ′,j′,i′)})

is

\begin{aligned}|\{(ξ, j) | 2|m̃_{ξ}(ν_{ξ})_{j}\}| & + |\{(ξ, j) | w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = \frac{1}{2}m̃_{ξ}(ν_{ξ})_{j}, 4|m̃_{ξ}(ν_{ξ})_{j}\}| \\ & + |\{\{(ξ, j) ≠ w̄_{f}(ξ, j)\} | 2 ∤ m̃_{ξ}(ν_{ξ})_{j}\}|. \end{aligned}

(The GL_{2} factors corresponding to (ξ, j) such that w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j} are split precisely when 4|m̃_{ξ}(ν_{ξ})_{j}, and those corresponding to \{(ξ, j) ≠ w̄_{f}(ξ, j)} are split precisely when 2|m̃_{ξ}(ν_{ξ})_{j}.) So Lusztig’s formula becomes

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩

= ∑_{f∈T^{F} \Θ_{T,λ}^{F}/K^{F}} (−1) \begin{matrix}|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j},4|m̃_{ξ}(ν_{ξ})_{j}\}| \\ +|\{\{(ξ,j)≠w̄_{f} (ξ,j)\} | 2∤m̃_{ξ}(ν_{ξ})_{j}\}| \end{matrix}

= ∑_{w ∈ Z′ with signed fixed points, signs stable under w_{F}, signature (n^{+}, n^{−})} (−1) \begin{matrix}|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j},4|m̃_{ξ}(ν_{ξ})_{j}\}| \\ + \frac{1}{2} |\{(ξ,j) | w̄(ξ,j)≠(ξ,j),2∤m̃_{ξ}(ν_{ξ})_{j}\}| \end{matrix} .

Using the same reasoning as in §3.1, we can transform this expression to get:

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩

= ∑_{involutions w_{1} ∈ Z_{S_{|ν_{1}|}} (ν_{1}) with signed fixed points, signs stable under ν_{1}, signature (n^{+}, n^{−}) + ℤ} (−1) \begin{matrix}|\{cycles of ν_{1} divisible by 4, fixed by w_{1}, not pointwise\}| \\ + \frac{1}{2} |\{odd cycles of ν_{1} moved by w_{1}\}| \end{matrix}

× ∑_{fixed-point free involutions w_{−1} ∈ Z_{S_{|ν_{−1}|}} (ν_{−1})} (−1) \begin{matrix}|\{cycles of ν_{−1} divisible by 4, fixed by w_{−1}\}| \\ + \frac{1}{2} |\{odd cycles of ν_{−1}\}| \end{matrix}

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ ξ^{−1}∈⟨σ̃⟩.ξ \\ ξ≠1,−1 \\ 4|m̃_{ξ} \end{matrix}} ( ∑_{involutions w_{ξ} ∈ Z_{S_{|ν_{ξ}|}} (ν_{ξ}) fixing no even cycles of ν_{ξ}} (−1)^{|\{cycles of ν_{ξ} fixed by w_{ξ}\}|})

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ ξ^{−1}∈⟨σ̃⟩.ξ \\ ξ≠1,−1 \\ 4∤m̃_{ξ} \end{matrix}} |\{ \begin{matrix}involutions w_{ξ} ∈ Z_{S_{|ν_{ξ}|}}(ν_{ξ}) \\ fixing no even cycles of ν_{ξ} \end{matrix} \}|

× ∏_{\begin{matrix}\{ξ_{1}≠ξ_{2}\}∈⟨σ̃⟩\L \\ ξ_{1}^{−1}∈⟨σ̃⟩.ξ_{2} \\ 2|m̃_{ξ_{1}} \end{matrix}} |\{ \begin{matrix}bijections \{1, ⋯, |ν_{ξ_{1}}|\} → \{1, ⋯, |ν_{ξ_{2}}|\} \\ intertwining ν_{ξ_{1}} and ν_{ξ_{2}} \end{matrix} \}|

× ∏_{\begin{matrix}\{ξ_{1}≠ξ_{2}\}∈⟨σ̃⟩\L \\ ξ_{1}^{−1}∈⟨σ̃⟩.ξ_{2} \\ 2∤m̃_{ξ_{1}} \end{matrix}}(−1)^{|\{odd cycles of ν_{ξ_{1}} \}|} |\{bijections ⋯ \}| .

So in addition to (3.1.4), we need the following combinatorial facts:

∑_{involutions w ∈ Z_{S_{|ν|}} (ν) with signed fixed points, signs stable under ν, signature (p^{+}, p^{−}) + ℤ} (−1)

|{cycles of ν divisible by 4, fixed by w, not pointwise}| + \frac{1}{2} |\{odd cycles of ν moved by w\}|

= ∑_{λ⊢|ν|}(−1)^{n(λ)}ā(p^{+}, p^{−}, λ′)χ_{ν}^{λ},

(3.3.2)

∑_{fixed-point free involutions w ∈ Z_{S_{|ν|}} (ν)} (−1)

|{cycles of ν divisible by 4, fixed by w}| + \frac{1}{2} |\{odd cycles of ν\}|

= (−1)^{\frac{|ν|}{2}} ∑_{\begin{matrix}λ⊢|ν| \\ λ′ even \end{matrix}} χ_{ν}^{λ}, and

(3.3.3)

(3.3.4)  |\{ \begin{matrix}involutions w ∈ Z_{S_{|ν|}}(ν) \\ fixing no even cycles of ν \end{matrix} \}| = ∑_{λ⊢|ν|} χ_{ν}^{λ}.

These will be proved in §5.

3.4. The U_{n}(𝔽_{q^{2}})/U_{n/2}(𝔽_{q^{4}}) Case. The final case to consider in this section is when F : G → G is a non-split Frobenius map for which F (s) = −s. Replacing s by a G^{F} -conjugate if necessary, we may assume that there is a form ⟨·, ·⟩ on V and an involution θ′ : G → G with the same properties as in §3.3. Then θ′F is the split Frobenius map induced by some F_{V} : V → V which respects ⟨·, ·⟩ and interchanges V^{+} and V^{−}. In particular, n is even, and n^{+} = n^{−} = \frac{n}{2}. One has G^{F} ≅ U_{n}(𝔽_{q^{2}}), K^{F} ≅ U_{n/2}(𝔽_{q^{4}}). The result is:

Theorem 3.4.1. For any \underline{ρ} ∈ P̂σ̃ ; n,

⟨χ\underline{ρ}, Ind_{U_{n/2}(𝔽_{q^{4}} )}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = \{ \begin{matrix}∏_{i}(m_{2i}(ρ′_{1}) + 1), & if ρ_{ξ} = ρ_{ξ^{−1}}, ∀ξ, 2|m_{2i+1}(ρ′_{1}), ∀i, and ρ′_{−1} is even \\ 0, & otherwise. \end{matrix}

By (1.3.2), it is equivalent to say that for any \underline{ν} ∈ P̂σ̃ ; n,

\begin{aligned}⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ & = ( ∑_{\begin{matrix}ρ_{1}⊢|ν_{1}| \\ 2|m_{2i+1}(ρ′_{1}) \end{matrix}}(−1)^{n(ρ_{1})}(∏_{i}(m_{2i}(ρ′_{1}) + 1))χ_{ν_{1}}^{ρ_{1}}) \\ & × ((−1)^{\frac{|ν_{−1}|}{2}} ∑_{\begin{matrix}ρ_{−1}⊢|ν_{−1}| \\ ρ′_{−1} even \end{matrix}} χ_{ν_{−1}}^{ρ_{−1}}) \\ & × ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ ξ^{−1}∈⟨σ̃⟩.ξ \\ ξ≠1,−1 \\ 4|m̃_{ξ} \end{matrix}}((−1)^{|ν_{ξ}|} ∑_{ρ_{ξ}⊢|ν_{ξ}|} χ_{ν_{ξ}}^{ρ_{ξ}}) ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ ξ^{−1}∈⟨σ̃⟩.ξ \\ ξ≠1,−1 \\ 4∤m̃_{ξ} \end{matrix}}( ∑_{ρ_{ξ}⊢|ν_{ξ}|} χ_{ν_{ξ}}^{ρ_{ξ}}) \\ & × ∏_{\begin{matrix}\{ξ_{1}≠ξ_{2}\}∈⟨σ̃⟩\L \\ ξ_{1}^{−1}∈⟨σ̃⟩.ξ_{2} \\ 2|m̃_{ξ_{1}} \end{matrix}} δ_{ν_{ξ_{1}} ν_{ξ_{2}}} z_{ν_{ξ_{1}}} ∏_{\begin{matrix}\{ξ_{1}≠ξ_{2}\}∈⟨σ̃⟩\L \\ ξ_{1}^{−1}∈⟨σ̃⟩.ξ_{2} \\ 2∤m̃_{ξ_{1}} \end{matrix}}(−1)^{|ν_{ξ_{1}} |}δ_{ν_{ξ_{1}} ν_{ξ_{2}}} z_{ν_{ξ_{1}}} . \end{aligned} (3.4.1)

For the signs here, see the comments after (3.3.1).

Let (T, λ), {L_{(ξ,j,i)}} and w_{F} be as in §3.3.

Lemma 3.4.2. For f ∈ Θ_{T}, w_{F (f)} = w_{F} w_{f}w_{F}^{−1}, and ϵ_{F (f)} = −ϵ_{f} ∘ w_{F}^{−1}.

Proof. This is an obvious modification of Lemma 3.3.2. □

Corollary 3.4.3. The map f ↦ (w_{f}, ϵ_{f}) induces a bijection

T^{F} \Θ_{T}^{F} /K^{F} →^{∼} \{ \begin{matrix}involutions in Z_{W (T)}(w_{F}) with signed fixed points, \\ signs reversed by w_{F} \end{matrix} \} .

Now Lemma 3.3.4 holds again here, with the same Z′ and exactly the same proof. So arguing as in §3.3, we get

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩

= ∑_{w ∈ Z′ with signed fixed points, signs reversed by w_{F}} (−1) \begin{matrix}|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j},4|m̃_{ξ}(ν_{ξ})_{j}\}| \\ + \frac{1}{2} |\{(ξ,j) | w̄(ξ,j)≠(ξ,j),2∤m̃_{ξ}(ν_{ξ})_{j}\}| \end{matrix} .

The rest of the proof is also the same as in §3.3, except that Equation (3.3.2) is replaced by:

∑_{involutions w ∈ Z_{S_{|ν|}} (ν) with signed fixed points, signs reversed by ν} (−1)

|{cycles of ν divisible by 4, fixed by w, not pointwise}| + \frac{1}{2} |\{odd cycles of ν moved by w\}|

= ∑_{\begin{matrix}λ⊢|ν| \\ 2|m_{2i+1}(λ′) \end{matrix}}(−1)^{n(λ)}(∏_{i}(m_{2i}(λ′) + 1))χ_{ν}^{λ}.

(3.4.2)

This will be proved in §5.

4. Cases where G/K = GL_{n}/O_{n} or GL_{n}/SO_{n}

In this section, we suppose that V has a nondegenerate symmetric form ⟨·, ·⟩, and that θ : G → G is the involution defined by

⟨θ(g)v, v′⟩ = ⟨v, g^{−1}v′⟩, ∀g ∈ G, v, v′ ∈ V.

Since G^{θ} = O(V, ⟨·, ·⟩) has two components, there are two possibilities for K. In order to treat them simultaneously, we will write K for G^{θ} and K^{∘} for (G^{θ})^{∘} = SO(V, ⟨·, ·⟩). So G/K ≅ GL_{n}/O_{n} and G/K^{∘} ≅ GL_{n}/SO_{n}.

Let T be a maximal torus of G with eigenlines L_{1}, …, L_{n}. If f ∈ Θ_{T}, then f^{−1}T f is a θ-stable maximal torus, so there is some w_{f} ∈ W (T) = S_{n} such that

(f^{−1}L_{i})^{⊥} = ⨁_{i′≠w_{f} (i)} f^{−1}L_{i′}, for all i,

and

θ|_{f^{−1}T f} = Ad(f^{−1}) ∘ w_{f} ∘ Ad(f) ∘ ι.

Since θ is an involution, w_{f} must be also; in contrast to §2, w_{f} need not be fixedpoint free, so

T ∩ fKf^{−1} = {t ∈ T | t|_{L_{i}} = ±1 if w_{f}(i) = i, t|_{L_{w_{f} (i)}} = (t|_{L_{i}})^{−1} if w_{f}(i) ≠ i}

is not necessarily connected.

Clearly the association f ↦ w_{f} induces a map

T \Θ_{T} /K → {involutions in W (T)}.

Proposition 4.0.1. This map is a bijection. Moreover, if w ∈ W (T) is an involution, the corresponding T –K double coset breaks into two T –K^{∘} double cosets if w is fixed-point free, and is a single T –K^{∘} double coset otherwise.

Proof. As with Propositions 2.0.1 and 3.0.1, this is better known as a statement about K-orbits on the flag variety (see [15, §6]). It is easy to prove. □

4.1. The GL_{n}(𝔽_{q})/O_{n}(𝔽_{q}) and GL_{n}(𝔽_{q})/SO_{n}(𝔽_{q}) Cases (n odd). In this subsection, suppose that n is odd and let F : G → G be a split Frobenius map which commutes with θ. So F is induced by a Frobenius map F_{V} on V which respects ⟨·, ·⟩, such that ⟨·, ·⟩ has Witt index ⌊\frac{n}{2}⌋ on V^{F_{V}}. One has G^{F} ≅ GL_{n}(𝔽_{q}), K^{F} ≅ O_{n}(𝔽_{q}), (K^{∘})^{F} ≅ SO_{n}(𝔽_{q}). The result is:

Theorem 4.1.1. For any \underline{ρ} ∈ P̂_{n}^{σ},

⟨χ\underline{ρ}, Ind_{O_{n}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ = \{ \begin{matrix}\frac{1}{2} ∏_{\begin{matrix}ξ∈⟨σ⟩\L \\ d_{ξ}=1 \end{matrix}}(∏_{i}(m_{i}(ρ_{ξ}) + 1)), & if d_{ξ} = −1 ⇒ ρ′_{ξ} is even \\ 0, & otherwise, and \end{matrix}

⟨χ\underline{ρ}, Ind_{SO_{n}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ = ⟨χ\underline{ρ}, Ind_{O_{n}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ + ⟨χ^{ζ.\underline{ρ}}, Ind_{O_{n}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩.

By (1.3.1), it is equivalent to say that for any \underline{ν} ∈ P̂_{n}^{σ},

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ = −\frac{1}{2} ∏_{\begin{matrix}ξ∈⟨σ⟩\L \\ d_{ξ}=1 \end{matrix}}((−1)^{|ν_{ξ}|} ∑_{ρ_{ξ}⊢|ν_{ξ}|}(∏_{i}(m_{i}(ρ_{ξ}) + 1))χ_{ν_{ξ}}^{ρ_{ξ}})

× ∏_{\begin{matrix}ξ∈⟨σ⟩\L \\ d_{ξ}=−1 \end{matrix}}( ∑_{\begin{matrix}ρ_{ξ}⊢|ν_{ξ}| \\ ρ′_{ξ} even \end{matrix}} χ_{ν_{ξ}}^{ρ_{ξ}}), and

⟨B\underline{ν}, Ind_{(K^{∘})^{F}}^{G^{F}} (1)⟩ = ⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ + ⟨B_{ζ.\underline{ν}}, Ind_{K^{F}}^{G^{F}} (1)⟩.

(4.1.1)

Here the sign in (1.3.1) has been distributed in an obvious way.

Let (T, λ) be a pair in the G^{F} -orbit corresponding to \underline{ν} ∈ P̂_{n}^{σ}. Define \{L_{(ξ,j,i)}\} and w_{F} exactly as in §2.1.

Lemma 4.1.2. For f ∈ Θ_{T}, w_{F (f)} = w_{F} w_{f}w_{F}^{−1}.

Proof. The proof is identical to that of Lemma 2.1.2. □

Corollary 4.1.3. The map f ↦ w_{f} induces a surjection

T^{F} \Θ_{T}^{F} /K^{F} ↠ {involutions in Z_{W (T)}(w_{F})}.

Proof. By Proposition 4.0.1 and Lemma 4.1.2, we may identify the set of involutions in Z_{W (T)}(w_{F}) with (T \Θ_{T} /K)^{F}. Under this identification, an involution is in the image of T^{F} \Θ_{T}^{F} /K^{F} precisely when the corresponding F -stable K-orbit on T\Θ_{T} contains an F -fixed point (by connectedness of T). Since n is odd, this K-orbit is a single K^{∘}-orbit, so this is automatic. □

For w an involution in Z_{W (T)}(w_{F}), we define w̄ and i(w, ξ, j) as in §2.1 and §3.1.

Lemma 4.1.4. For f ∈ Θ_{T}^{F}, the number of T^{F} –K^{F} double cosets in T fK is

2^{|{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0}|−1},

and each T^{F} –K^{F} double coset is a single T^{F} –(K^{∘})^{F} double coset.

Proof. Since the image of the Lang map on K is K^{∘}, the number of T^{F} –K^{F} double cosets in T fK is the same as the number of orbits of T ∩ fKf^{−1} on T ∩ fK^{∘}f^{−1} for the action t.t′ = tt′F (t)^{−1}. Since

\begin{aligned}T ∩ fKf^{−1} & = \{(α_{(ξ,j,i)}) ∈ ∏_{\begin{matrix}(ξ,j,i) \\ w_{f} (ξ,j,i)=(ξ,j,i) \end{matrix}}±1\} \\ & × \{(β_{(ξ,j,i)}) ∈ ∏_{\begin{matrix}(ξ,j,i) \\ w_{f} (ξ,j,i)≠(ξ,j,i) \end{matrix}}k^{×} | β_{w_{f} (ξ,j,i)} = β_{(ξ,j,i)}^{−1} \} , \end{aligned}

this is the same as the number of orbits of

\{(ϵ_{(ξ,j,i)}) ∈ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}}(±1)^{m_{ξ}(ν_{ξ})_{j}} \}

on

\{(ϵ′_{(ξ,j,i)}) ∈ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}}(±1)^{m_{ξ}(ν_{ξ})_{j}} | ∏_{(ξ,j,i)} ϵ′_{(ξ,j,i)} = 1\}

for the action (ϵ_{(ξ,j,i)}).(ϵ′_{(ξ,j,i)}) = (ϵ_{(ξ,j,i)})(ϵ_{(ξ,j,i−1)})(ϵ′_{(ξ,j,i)}). Since n is odd, there must be some (ξ, j) for which w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = 0, 2 ∤ m_{ξ}(ν_{ξ})_{j}. The first statement follows. Modifying the above analysis for K^{∘}, we get the second statement. □

Now let Z′ be the set of involutions w in Z_{W (T)}(w_{F}) for which w̄(ξ, j) = (ξ, j′) (for some j′) and m_{ξ} | i(w, ξ, j) hold for all (ξ, j), and w̄(ξ, j) = (ξ, j), i(w, ξ, j) = 0, d_{ξ} = −1 implies 2 | (ν_{ξ})_{j}. Define Z′′ in the same way but with the last condition changed to: w̄(ξ, j) = (ξ, j), i(w, ξ, j) = 0, d_{ζξ} = −1 implies 2 | (ν_{ξ})_{j}.

Lemma 4.1.5. For f ∈ Θ_{T}^{F}, f ∈ Θ_{T,K,λ}^{F} ⇔ w_{f} ∈ Z′, and f ∈ Θ_{T,K^{∘},λ}^{F} ⇔ w_{f} ∈ Z′ ∪ Z′′.

Proof. We first note that ϵ_{T,f} = 1, as may be seen directly (using formulas for 𝔽_{q}-rank such as those below) or deduced by the method of [11, Lemma 11.3]. So

\begin{aligned}Θ_{T,K,λ}^{F} = \{f ∈ Θ_{T}^{F} | & λ|_{(T ∩fKf^{−1})^{F}} = 1\} \\ & ⊆ Θ_{T,K^{∘},λ}^{F} = \{f ∈ Θ_{T}^{F} | λ|_{(T ∩fK^{∘}f^{−1})^{F}} = 1\}. \end{aligned}

Now

(T ∩ fKf^{−1})^{F} ≅ {(α_{(ξ,j)}) ∈ ∏_{(ξ,j)}(k^{×})^{σ^{m_{ξ}(ν_{ξ})_{j}}} |σ^{i(w_{f} ,ξ,j)}(α_{w̄_{f} (ξ,j)}) = α_{(ξ,j)}^{−1}, ∀ξ, j }

= \{(α_{(ξ,j)}) ∈ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}}(±1)\}

× \{(β_{(ξ,j)}) ∈ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)= \frac{1}{2} m_{ξ}(ν_{ξ})_{j} \end{matrix}}(k^{×})^{ισ^{\frac{1}{2} m_{ξ}(ν_{ξ})_{j}}} \}

× \{(γ_{(ξ,j)}) ∈ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)≠(ξ,j) \end{matrix}}(k^{×})^{σ^{m_{ξ}(ν_{ξ})_{j}}} |γ_{w̄_{f} (ξ,j)} = σ^{−i(w_{f} ,ξ,j)}(γ_{(ξ,j)}^{−1})\}

and in this form λ|_{(T ∩fKf^{−1})^{F}} is the function

((α_{(ξ,j)}), (β_{(ξ,j)}), (γ_{(ξ,j)})) ↦ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w,ξ,j)=0 \end{matrix}}⟨α_{(ξ,j)}, ξ⟩^{σ^{m_{ξ}(ν_{ξ})_{j}}}

× ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)= \frac{1}{2} m_{ξ}(ν_{ξ})_{j} \end{matrix}}⟨β_{(ξ,j)}, ξ⟩^{σ^{m_{ξ}(ν_{ξ})_{j}}}

× ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)≠(ξ,j) \end{matrix}}⟨γ_{(ξ,j)}, ξ⟩^{σ^{m_{ξ}(ν_{ξ})_{j}}} .

Hence λ|_{(T ∩fKf^{−1})^{F}} = 1 if and only if:

• whenever w̄_{f}(ξ, j) = (ξ, j) and i(w_{f}, ξ, j) = 0, we have ⟨−1, ξ⟩^{σ^{m_{ξ}(ν_{ξ})_{j}}} = 1;

• whenever w̄_{f}(ξ, j) = (ξ, j) and i(w_{f}, ξ, j) = \frac{1}{2}m_{ξ}(ν_{ξ})_{j}, σ^{\frac{1}{2} m_{ξ}(ν_{ξ})_{j}}(ξ) = ξ; and

• whenever w̄_{f}(ξ, j) = (ξ′, j′) ≠ (ξ, j), ξσ^{−i(w_{f} ,ξ,j)}(ξ′)^{−1} = 1. By definition of m_{ξ} and d_{ξ}, this is equivalent to w_{f} ∈ Z′. (T ∩ fK^{∘}f^{−1})^{F} is the same as (T ∩ fKf^{−1})^{F} except that the condition

∏_{(ξ,j)} α_{(ξ,j)}^{m_{ξ}(ν_{ξ})_{j}} = 1

is imposed. So the second statement follows similarly, using the fact that

d_{ζξ} = −1 ⇔ d_{ξ} = −(−1)^{m_{ξ}}.

□

Corollary 4.1.6. The map f ↦ w_{f} induces surjections T^{F}\Θ_{T,K,λ}^{F}/K^{F} ↠ Z′ and T^{F} \Θ_{T,K^{∘},λ}^{F}/(K^{∘})^{F} ↠ Z′ ∪ Z′′.

Now the 𝔽_{q}-rank of T is |{(ξ, j)}|, and that of

Z_{G}((T ∩ fKf^{−1})^{∘}) = Z_{G}((T ∩ fK^{∘}f^{−1})^{∘})

= GL ( ⨁_{\begin{matrix}(ξ,j,i) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}} L_{(ξ,j,i)})

× ∏_{\begin{matrix}(ξ,j,i) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)= \frac{1}{2} m_{ξ}(ν_{ξ})_{j} \end{matrix}} GL(L_{(ξ,j,i)}) × ∏_{\begin{matrix}(ξ,j,i) \\ w̄_{f} (ξ,j)≠(ξ,j) \end{matrix}} GL(L_{(ξ,j,i)})

is

\begin{aligned}|\{(ξ, j)\}| & + ∑_{\begin{matrix}(ξ,j,i) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}}(m_{ξ}(ν_{ξ})_{j} − 1) \\ & ≡ |\{(ξ, j)\}| + n + |\{(ξ, j) | w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = 0\}| mod 2. \end{aligned}

So Lusztig’s formula gives

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ = − ∑_{f∈T^{F} \Θ_{T,K,λ}^{F}/K^{F}} (−1)^{|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0\}|}

= − ∑_{w∈Z′}(−1)^{|{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0}|}2^{|{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0}|−1}

= −\frac{1}{2} ∑_{w∈Z′}(−2)^{|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0\}|}, and

⟨B\underline{ν}, Ind_{(K^{∘})^{F}}^{G^{F}} (1)⟩

= −\frac{1}{2} ∑_{w∈Z′}(−2)^{|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0\}|}

− \frac{1}{2} ∑_{w∈Z′′}(−2)^{|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0\}|},

since the fact that n is odd implies that Z′ and Z′′ are disjoint.

For any w ∈ Z′∪Z′′, we define a permutation ŵ as in §2.1. It is clear that w ↦ ŵ defines bijections between Z′ and the set of involutions ŵ in ∏_{ξ∈⟨σ⟩\L} Z_{S_{|ν_{ξ}|}}(ν_{ξ}) such that ŵ(ξ, j, s) = (ξ, j, s) and d_{ξ} = −1 implies 2 | (ν_{ξ})_{j}, and between Z′′ and the set of involutions ŵ such that ŵ(ξ, j, s) = (ξ, j, s) and d_{ζξ} = −1 implies 2 | (ν_{ξ})_{j}. Hence

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩

= −\frac{1}{2} ∏_{ξ∈⟨σ⟩\L} ( ∑_{involutions w_{ξ} ∈ Z_{S_{|ν_{ξ}|}} (ν_{ξ}), fixing no odd cycles of ν_{ξ} if d_{ξ} = −1} (−2)^{|\{cycles of ν_{ξ} fixed by w_{ξ} pointwise\}|})

and

⟨B\underline{ν}, Ind_{(K^{∘})^{F}}^{G^{F}} (1)⟩ = ⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ + ⟨B_{ζ.\underline{ν}}, Ind_{K^{F}}^{G^{F}} (1)⟩.

So to prove (4.1.1), the combinatorial facts we need are:

∑_{involutions w ∈ Z_{S|ν|} (ν)}

(−2)^{|{cycles of ν fixed by w pointwise}|}

= (−1)^{|ν|} ∑_{λ⊢|ν|}(∏_{i}(m_{i}(λ) + 1))χ_{ν}^{λ}, and

(4.1.2)

(4.1.3)  ∑_{involutions w ∈ Z_{S_{|ν|}} (ν) fixing no odd cycles of ν} (−2)^{|\{cycles of ν fixed by w pointwise\}|} = ∑_{\begin{matrix}λ⊢|ν| \\ λ′ even \end{matrix}} χ_{ν}^{λ}.

These will be proved in §5.

4.2. The GL_{n}(𝔽_{q})/O_{n}^{±}(𝔽_{q}) and GL_{n}(𝔽_{q})/SO_{n}^{±}(𝔽_{q}) Cases (n even). In this subsection, suppose that n is even and F : G → G is a split Frobenius map which commutes with θ. So F is induced by a Frobenius map F_{V} on V which respects ⟨·, ·⟩, and once again G^{F} ≅ GL_{n}(𝔽_{q}). The Witt index of ⟨·, ·⟩ on V^{F_{V}} is either \frac{n}{2} or \frac{n}{2} − 1, and accordingly either K^{F} ≅ O_{n}^{+}(𝔽_{q}), (K^{∘})^{F} ≅ SO_{n}^{+}(𝔽_{q}) or K^{F} ≅ O_{n}^{−}(𝔽_{q}), (K^{∘})^{F} ≅ SO_{n}^{−}(𝔽_{q}). Let ϵ be 1 in the first case and −1 in the second case.

Theorem 4.2.1. For any \underline{ρ} ∈ P̂_{n}^{σ},

⟨χ\underline{ρ}, Ind_{O_{n}^{ϵ}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ = \{ \begin{matrix}\frac{1}{2} ∏_{\begin{matrix}ξ∈⟨σ⟩\L \\ d_{ξ}=1 \end{matrix}}(∏_{i}(m_{i}(ρ_{ξ}) + 1)), & if d_{ξ} = −1 ⇒ ρ′_{ξ} is even \\ 0, & otherwise \end{matrix}

+ \{ \begin{matrix}\frac{1}{2}ϵ, & if all ρ′_{ξ} are even \\ 0, & otherwise, and \end{matrix}
⟨χ\underline{ρ}, Ind_{SO_{n}^{ϵ}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ = ⟨χ\underline{ρ}, Ind_{O_{n}^{ϵ}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩ + ⟨χ^{ζ.\underline{ρ}}, Ind_{O_{n}^{ϵ}(𝔽_{q})}^{GL_{n}(𝔽_{q})}(1)⟩.

By (1.3.1), it is equivalent to say that for any \underline{ν} ∈ P̂_{n}^{σ},

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ = \frac{1}{2} ∏_{\begin{matrix}ξ∈⟨σ⟩\L \\ d_{ξ}=1 \end{matrix}}((−1)^{|ν_{ξ}|} ∑_{ρ_{ξ}⊢|ν_{ξ}|}(∏_{i}(m_{i}(ρ_{ξ}) + 1))χ_{ν_{ξ}}^{ρ_{ξ}})

× ∏_{\begin{matrix}ξ∈⟨σ⟩\L \\ d_{ξ}=−1 \end{matrix}}( ∑_{\begin{matrix}ρ_{ξ}⊢|ν_{ξ}| \\ ρ′_{ξ} even \end{matrix}} χ_{ν_{ξ}}^{ρ_{ξ}})

+ \frac{1}{2}ϵ ∏_{ξ∈⟨σ⟩\L}( ∑_{\begin{matrix}ρ_{ξ}⊢|ν_{ξ}| \\ ρ′_{ξ} even \end{matrix}} χ_{ν_{ξ}}^{ρ_{ξ}}), and

⟨B\underline{ν}, Ind_{(K^{∘})^{F}}^{G^{F}} (1)⟩ = ⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ + ⟨B_{ζ.\underline{ν}}, Ind_{K^{F}}^{G^{F}} (1)⟩.

(4.2.1)

Let (T, λ) be a pair in the G^{F} -orbit corresponding to \underline{ν} ∈ P̂_{n}^{σ}. Define \{L_{(ξ,j,i)}\} and w_{F} exactly as in §2.1.

Lemma 4.2.2. For f ∈ Θ_{T}, w_{F (f)} = w_{F} w_{f}w_{F}^{−1}.

Proof. As in the previous subsection, the proof of Lemma 2.1.2 works. □

Corollary 4.2.3. The map f ↦ w_{f} induces a map

T^{F} \Θ_{T}^{F} /K^{F} → {involutions in Z_{W (T)}(w_{F})}.

If sign(w_{F}) = ϵ, this map is surjective. If sign(w_{F}) = −ϵ, the image consists of all involutions with at least one fixed point.

Proof. As in the proof of Corollary 4.1.3, an involution is in the image of T^{F}\Θ_{T}^{F} /K^{F} precisely when the corresponding F -stable K-orbit on T \Θ_{T} contains an F -fixed point. If the involution has a fixed point, this K-orbit is a single K^{∘}-orbit, so this is automatic. Suppose the involution w ∈ Z_{W (T)}(w_{F}) is fixed-point free. It is in the image of T^{F} \Θ_{T}^{F} /K^{F} precisely when there exists a decomposition of V into lines {L′_{i} | 1 ≤ i ≤ n} such that

(1) (L′_{i})^{⊥} = ⊕_{i′≠w(i)}L′_{i′}, and

(2) F_{V} (L′_{i}) = L′_{w_{F} (i)}

(here we have identified W (T) with S_{n} in some way). We must prove that this happens if and only if sign(w_{F}) = ϵ. Since ϵ is multiplicative with respect to F_{V} - stable orthogonal direct sums, we may assume that ⟨w⟩ × ⟨w_{F} ⟩ acts transitively on {1, ⋯, n}. So if sign(w_{F}) = 1, then ⟨w_{F} ⟩ has two orbits on {1, ⋯, n} which w interchanges; in this case the existence of {L′_{i}} as above is clearly equivalent to the existence of a decomposition V = V_{1} ⊕ V_{2} into F_{V} -stable Lagrangian subspaces, which indeed happens if and only if ϵ = 1. On the other hand, if sign(w_{F}) = −1, then w_{F} is an n-cycle and w = w_{F}^{n/2}; in this case the existence of {L′_{i}} as above is easily seen to be equivalent to the existence of a Lagrangian subspace V_{1} of V such that dim V_{1} ∩ F_{V} (V_{1}) = \frac{n}{2} − 1, which indeed happens if and only if ϵ = −1. □

For w an involution in Z_{W (T)}(w_{F}), we define w̄ and i(w, ξ, j) as usual.

Lemma 4.2.4. For f ∈ Θ_{T}^{F}, the number of T^{F} –K^{F} double cosets in T fK is

1,

if w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = 0 never occurs

2^{|{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0}|−1}, otherwise,

and the number of T^{F} –(K^{∘})^{F} double cosets in T fK^{∘} is

2^{|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0\}|},  \begin{matrix}if w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = 0 \\ ⇒ 2 | m_{ξ}(ν_{ξ})_{j} \end{matrix}
2^{|{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0}|−1}, otherwise.

Proof. The method of proof of Lemma 4.1.4 applies again here. □

Now define Z′ and Z′′ in exactly the same way as in §4.1.

Lemma 4.2.5. For f ∈ Θ_{T}^{F}, f ∈ Θ_{T,K,λ}^{F} ⇔ w_{f} ∈ Z′, and f ∈ Θ_{T,K^{∘},λ}^{F} ⇔ w_{f} ∈ Z′ ∪ Z′′.

Proof. The proof is exactly the same as that of Lemma 4.1.5. □

Corollary 4.2.6. (1) The map f ↦ w_{f} induces maps T^{F} \Θ_{T,K,λ}^{F}/K^{F} → Z′ and T^{F} \ Θ_{T,K^{∘},λ}^{F}/(K^{∘})^{F} → Z′ ∪ Z′′. If sign(w_{F}) = ϵ, these maps are surjective; if sign(w_{F}) = −ϵ, their images are Z′ \ Z′′′ and (Z′ ∪ Z′′) \ Z′′′, where Z′′′ is the set of fixed-point free involutions in Z′.

(2) If w is in the image of T^{F} \Θ_{T,K,λ}^{F}/K^{F} → Z′, there are

\{ \begin{matrix}1, & if w ∈ Z′′′ \\ 2^{|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0\}|−1}, & otherwise \end{matrix}

T^{F} –K^{F} double cosets in the preimage of w.

(3) If w is in the image of T^{F} \Θ_{T,K^{∘},λ}^{F}/(K^{∘})^{F} → Z′ ∪ Z′′, there are

2,  if w ∈ Z′′′
2^{|{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0}|}, if w ∈ (Z′ ∩ Z′′) \ Z′′′
2^{|{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0}|−1}, otherwise

T^{F} –(K^{∘})^{F} double cosets in the preimage of w.

Proof. This follows by combining Proposition 4.0.1, Corollary 4.2.3, Lemma 4.2.4 and Lemma 4.1.5. Note that Z′ ∩ Z′′ is precisely the set of w ∈ Z′ ∪ Z′′ such that w̄(ξ, j) = (ξ, j), i(w, ξ, j) = 0 ⇒ 2 | m_{ξ}(ν_{ξ})_{j}. □

Now as in §4.1, the 𝔽_{q}-rank of T is |{(ξ, j)}|, and that of Z_{G}((T ∩ fKf^{−1})^{∘}) = Z_{G}((T ∩ fK^{∘}f^{−1})^{∘}) is congruent to

|{(ξ, j)}| + n + |{(ξ, j) | w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = 0}| mod 2.

32 ANTHONY HENDERSON

So Lusztig’s formula gives

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ = ∑_{f∈T^{F} \Θ_{T,K,λ}^{F}/K^{F}} (−1)^{|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0\}|}

= ∑_{w∈Z′}(−1)^{|{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0}|}

× \{ \begin{matrix}1, & if w ∈ Z′′′ \\ 2^{|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0\}|−1}, & otherwise \end{matrix}

− \frac{1}{2}(1 − sign(w_{F})ϵ)|Z′′′|

= \frac{1}{2} ∑_{w∈Z′}(−2)^{|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0\}|}

+ \frac{1}{2}(−1)^{|\{(ξ,j) | 2|m_{ξ}(ν_{ξ})j\}|}ϵ|Z′′′|, and

⟨B\underline{ν}, Ind_{(K^{∘})^{F}}^{G^{F}} (1)⟩

= ∑_{w∈Z′∪Z′′}(−1)^{|{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0}|}

× \{ \begin{matrix}2, & if w ∈ Z′′′ \\ 2^{|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0\}|}, & if w ∈ (Z′ ∩ Z′′) \ Z′′′ \\ 2^{|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0\}|−1}, & otherwise \end{matrix}

− \frac{1}{2}(1 − sign(w_{F})ϵ)2|Z′′′|

= \frac{1}{2} ∑_{w∈Z′}(−2)^{|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0\}|}

+ \frac{1}{2} ∑_{w∈Z′′}(−2)^{|\{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0\}|}

+ (−1)^{|{(ξ,j) | 2|m_{ξ}(ν_{ξ})j}|}ϵ|Z′′′|.

We transform these expressions as in the previous subsection to obtain:

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩

= \frac{1}{2} ∏_{ξ∈⟨σ⟩\L} ( ∑_{involutions w_{ξ} ∈ Z_{S_{|ν_{ξ}|}} (ν_{ξ}), fixing no odd cycles of ν_{ξ} if d_{ξ} = −1} (−2)^{|\{cycles of ν_{ξ} fixed by w_{ξ} pointwise\}|})

+ \frac{1}{2}ϵ ∏_{ξ∈⟨σ⟩\L} ϵ_{ν_{ξ}} |\{ \begin{matrix}fixed-point free \\ involutions in Z_{S_{|ν_{ξ}|}}(ν_{ξ}) \end{matrix} \}| ,

and

⟨B\underline{ν}, Ind_{(K^{∘})^{F}}^{G^{F}} (1)⟩ = ⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ + ⟨B_{ζ.\underline{ν}}, Ind_{K^{F}}^{G^{F}} (1)⟩.

Here we have used the fact that if there exists a fixed-point free involution in Z_{S_{|ν_{ξ}|}}(ν_{ξ}), then |{odd cycles of ν_{ξ}}| is even, so

(−1)^{|{cycles of ν_{ξ}}|} = (−1)^{|{even cycles of ν_{ξ}}|} = ϵ_{ν_{ξ}}.

So along with (4.1.2) and (4.1.3), we need (2.1.2) multiplied on both sides by ϵ_{ν}.

4.3. The U_{n}(𝔽_{q^{2}})/O_{n}(𝔽_{q}) and U_{n}(𝔽_{q^{2}})/SO_{n}(𝔽_{q}) Cases (n odd). In this subsection, suppose that n is odd and take F : G → G to be a non-split Frobenius map commuting with θ. Thus θF is a split Frobenius map induced by F_{V} as in §4.1. One has G^{F} ≅ U_{n}(𝔽_{q^{2}}), K^{F} ≅ O_{n}(𝔽_{q}), (K^{∘})^{F} ≅ SO_{n}(𝔽_{q}). The result is:

Theorem 4.3.1. For any \underline{ρ} ∈ P̂σ̃ ; n,

⟨χ\underline{ρ}, Ind_{O_{n}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = \{ \frac{1}{2} ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=1 \\ 2∤m̃_{ξ} \end{matrix}} (∏_{i}(m_{2i+1}(ρ_{ξ}) + 1)) ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=−1 \\ 2∤m̃_{ξ} \end{matrix}}(∏_{i}(m_{2i}(ρ_{ξ}) + 1)) × ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=1 \\ 2|m̃_{ξ} \end{matrix}}(∏_{i}(m_{i}(ρ_{ξ}) + 1)), if d̃_{ξ} = 1, 2 ∤ m̃_{ξ} ⇒ 2|m_{2i}(ρ_{ξ}), ∀i, d̃_{ξ} = −1, 2 ∤ m̃_{ξ} ⇒ 2|m_{2i+1}(ρ_{ξ}), ∀i, and d̃_{ξ} = −1, 2|m̃_{ξ} ⇒ ρ′_{ξ} is even 0, otherwise, and

⟨χ\underline{ρ}, Ind_{SO_{n}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = ⟨χ\underline{ρ}, Ind_{O_{n}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ + ⟨χ^{ζ̃.\underline{ρ}}, Ind_{O_{n}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩.

By (1.3.2), it is equivalent to say that for any \underline{ν} ∈ P̂σ̃ ; n,

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ = \frac{1}{2}(−1)^{⌊ \frac{n}{2} ⌋} ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=1 \\ 2∤m̃_{ξ} \end{matrix}} ( ∑_{\begin{matrix}ρ_{ξ}⊢|ν_{ξ}| \\ 2|m_{2i}(ρ_{ξ}) \end{matrix}}(−1)^{n(ρ′_{ξ})}(∏_{i}(m_{2i+1}(ρ_{ξ}) + 1))χ_{ν_{ξ}}^{ρ_{ξ}} )

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=−1 \\ 2∤m̃_{ξ} \end{matrix}} ( ∑_{\begin{matrix}ρ_{ξ}⊢|ν_{ξ}| \\ 2|m_{2i+1}(ρ_{ξ}) \end{matrix}}(−1)^{n(ρ′_{ξ})}(∏_{i}(m_{2i}(ρ_{ξ}) + 1))χ_{ν_{ξ}}^{ρ_{ξ}} )

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=1 \\ 2|m̃_{ξ} \end{matrix}}(−1)^{|ν_{ξ}|} ( ∑_{ρ_{ξ}⊢|ν_{ξ}|}(∏_{i}(m_{i}(ρ_{ξ}) + 1))χ_{ν_{ξ}}^{ρ_{ξ}} )

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=−1 \\ 2|m̃_{ξ} \end{matrix}} ( ∑_{\begin{matrix}ρ_{ξ}⊢|ν_{ξ}| \\ ρ′_{ξ} even \end{matrix}} χ_{ν_{ξ}}^{ρ_{ξ}} ) , and

⟨B\underline{ν}, Ind_{(K^{∘})^{F}}^{G^{F}} (1)⟩ = ⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ + ⟨B_{ζ̃.\underline{ν}}, Ind_{K^{F}}^{G^{F}} (1)⟩.

(4.3.1)

34 ANTHONY HENDERSON

Here the sign

(−1)^{⌈ \frac{n}{2} ⌉+∑_{ξ∈⟨σ̃⟩\L} m̃_{ξ}n(ρ′_{ξ})+|ρ_{ξ}|} = (−1)^{⌊ \frac{n}{2} ⌋+∑_{ξ∈⟨σ̃⟩\L} m̃_{ξ}(n(ρ′_{ξ})+|ρ_{ξ}|)+|ρ_{ξ}|} of (1.3.2) has been distributed in an obvious way.

The proof of these statements is mostly very similar to that of (4.1.1). Let (T, λ) be a pair in the G^{F} -orbit corresponding to \underline{ν} ∈ P̂σ̃ ; n. Define \{L_{(ξ,j,i)}\} and w_{F} exactly as in §2.2. Lemma 4.3.2. For f ∈ Θ_{T}, w_{F (f)} = w_{F} w_{f}w_{F}^{−1}. Proof. The proof is identical to that of Lemma 2.2.2. □ Corollary 4.3.3. The map f ↦ w_{f} induces a surjection

T^{F} \Θ_{T}^{F} /K^{F} ↠ {involutions in Z_{W (T)}(w_{F})}.

Proof. This is deduced in the same way as Corollary 4.1.3. □

For w an involution in Z_{W (T)}(w_{F}), define w̄ and i(w, ξ, j) exactly as in §2.1, but with σ̃ instead of σ, m̃_{ξ} instead of m_{ξ}. Lemma 4.3.4. For f ∈ Θ_{T}^{F}, the number of T^{F} –K^{F} double cosets in T fK is

2^{|{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0}|−1},

and each T^{F} –K^{F} double coset is a single T^{F} –(K^{∘})^{F} double coset. Proof. The proof is the same as that of 4.1.4. □

Now define Z′ and Z′′ as in §4.1, but with σ̃ instead of σ, m̃_{ξ} instead of m_{ξ}, and d̃_{ξ} instead of d_{ξ}. Lemma 4.3.5. For f ∈ Θ_{T}^{F}, f ∈ Θ_{T,K,λ}^{F} ⇔ w_{f} ∈ Z′, and f ∈ Θ_{T,K^{∘},λ}^{F} ⇔ w_{f} ∈ Z′ ∪ Z′′. Proof. We have

(T ∩ fKf^{−1})^{F} ≅ \{(α_{(ξ,j)}) ∈ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}}(±1)\}

× \{(β_{(ξ,j)}) ∈ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j} \end{matrix}}(k^{×})^{ισ̃^{\frac{1}{2} m̃_{ξ}(ν_{ξ})_{j}}} \}

× \{(γ_{(ξ,j)}) ∈ ∏_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)≠(ξ,j) \end{matrix}}(k^{×})^{σ^{m̃_{ξ}(ν_{ξ})_{j}}} |γ_{w̄_{f} (ξ,j)} = σ^{−i(w_{f} ,ξ,j)}(γ_{(ξ,j)}^{−1})\}

and λ|_{(T ∩fKf^{−1})^{F}} has the usual form (cf. the proof of Lemma 4.1.5). Now the 𝔽_{q}-rank of

Z_{G}((T ∩ fKf^{−1})^{∘}) = Z_{G}((T ∩ fK^{∘}f^{−1})^{∘})

= GL ( ⨁_{\begin{matrix}(ξ,j,i) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}} L_{(ξ,j,i)})

× ∏_{\begin{matrix}(ξ,j,i) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j} \end{matrix}} GL(L_{(ξ,j,i)}) × ∏_{\begin{matrix}(ξ,j,i) \\ w̄_{f} (ξ,j)≠(ξ,j) \end{matrix}} GL(L_{(ξ,j,i)})

is

\begin{aligned}⌊\frac{1}{2} ∑_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}} m̃_{ξ}(ν_{ξ})_{j}⌋ & + |\{(ξ, j) | w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = \frac{1}{2}m̃_{ξ}(ν_{ξ})_{j}\}| \\ & + |\{(ξ, j) | w̄_{f}(ξ, j) ≠ (ξ, j), 2|m̃_{ξ}(ν_{ξ})_{j}\}|. \end{aligned}

If t ∈ (T ∩ fKf^{−1})^{F} corresponds to ((α_{(ξ,j)}), (β_{(ξ,j)}), (γ_{(ξ,j)})), then

Z_{G}^{∘}(t) ∩ Z_{G}((T ∩ fKf^{−1})^{∘})

= GL ( ⨁_{\begin{matrix}(ξ,j,i) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \\ α_{(ξ,j)}=1 \end{matrix}} L_{(ξ,j,i)}) × GL ( ⨁_{\begin{matrix}(ξ,j,i) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \\ α_{(ξ,j)}=−1 \end{matrix}} L_{(ξ,j,i)})

× ∏_{\begin{matrix}(ξ,j,i) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j} \end{matrix}} GL(L_{(ξ,j,i)}) × ∏_{\begin{matrix}(ξ,j,i) \\ w̄_{f} (ξ,j)≠(ξ,j) \end{matrix}} GL(L_{(ξ,j,i)})

has 𝔽_{q}-rank which differs from that of Z_{G}((T ∩ fKf^{−1})^{∘}) by

⌊\frac{1}{2} ∑_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \\ α_{(ξ,j)}=1 \end{matrix}} m̃_{ξ}(ν_{ξ})_{j}⌋ + ⌊\frac{1}{2} ∑_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \\ α_{(ξ,j)}=−1 \end{matrix}} m̃_{ξ}(ν_{ξ})_{j}⌋ − ⌊\frac{1}{2} ∑_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}} m̃_{ξ}(ν_{ξ})_{j}⌋.

Since n is odd,

∑_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}} m̃_{ξ}(ν_{ξ})_{j}

is odd, so this difference is zero. Thus ϵ_{T,f} = 1. The rest of the proof follows that of Lemma 4.1.5. □

36 ANTHONY HENDERSON

Corollary 4.3.6. The map f ↦ w_{f} induces surjections T^{F}\Θ_{T,K,λ}^{F}/K^{F} ↠ Z′ and T^{F} \Θ_{T,K^{∘},λ}^{F}/(K^{∘})^{F} ↠ Z′ ∪ Z′′.

Now the 𝔽_{q}-rank of T is |{(ξ, j) | 2|m̃_{ξ}(ν_{ξ})_{j}}|, and that of Z_{G}((T ∩ fKf^{−1})^{∘}) is given above, whence

𝔽_{q}-rank(T) + 𝔽_{q}-rank(Z_{G}((T ∩ fKf^{−1})^{∘}))

≡ ⌊\frac{1}{2} ∑_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}} m̃_{ξ}(ν_{ξ})_{j}⌋

+ |{(ξ, j) | w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = 0, 2|m̃_{ξ}(ν_{ξ})_{j}}|

≡ ⌊\frac{n}{2}⌋ + |\{(ξ, j) | w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = 0, 2|m̃_{ξ}(ν_{ξ})_{j}\}|

+ |\{(ξ, j) | w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = \frac{1}{2}m̃_{ξ}(ν_{ξ})_{j}, 4 ∤ m̃_{ξ}(ν_{ξ})_{j}\}|

+ \frac{1}{2}|\{(ξ, j) | w̄_{f}(ξ, j) ≠ (ξ, j), 2 ∤ m̃_{ξ}(ν_{ξ})_{j}\}| mod 2.

So Lusztig’s formula gives (compare §4.1):

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩

= \frac{1}{2}(−1)^{⌊ \frac{n}{2} ⌋} ∑_{w∈Z′}(−1) \begin{matrix}|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0,2|m̃_{ξ}(ν_{ξ})_{j}\}| \\ +|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j},4∤m̃_{ξ}(ν_{ξ})_{j}\}| \\ + \frac{1}{2} |\{(ξ,j) | w̄_{f} (ξ,j)≠(ξ,j),2∤m̃_{ξ}(ν_{ξ})_{j}\}| \end{matrix}

× 2^{|{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0}|}, and

⟨B\underline{ν}, Ind_{(K^{∘})^{F}}^{G^{F}} (1)⟩

= \frac{1}{2}(−1)^{⌊ \frac{n}{2} ⌋} ∑_{w∈Z′}(−1) \begin{matrix}|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0,2|m̃_{ξ}(ν_{ξ})_{j}\}| \\ +|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j},4∤m̃_{ξ}(ν_{ξ})_{j}\}| \\ + \frac{1}{2} |\{(ξ,j) | w̄_{f} (ξ,j)≠(ξ,j),2∤m̃_{ξ}(ν_{ξ})_{j}\}| \end{matrix}

× 2^{|{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0}|}

+ \frac{1}{2}(−1)^{⌊ \frac{n}{2} ⌋} ∑_{w∈Z′′}(−1) \begin{matrix}|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0,2|m̃_{ξ}(ν_{ξ})_{j}\}| \\ +|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j},4∤m̃_{ξ}(ν_{ξ})_{j}\}| \\ + \frac{1}{2} |\{(ξ,j) | w̄_{f} (ξ,j)≠(ξ,j),2∤m̃_{ξ}(ν_{ξ})_{j}\}| \end{matrix}

× 2^{|{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0}|},

since Z′ and Z′′ are disjoint by the oddness of n.

K^{F} -INVARIANTS IN IRREDUCIBLE REPRESENTATIONS OF G^{F}, WHEN G = GL_{n} 37

As in §4.1, we transform these expressions to get

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩

= \frac{1}{2}(−1)^{⌊ \frac{n}{2} ⌋}

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ 2∤m̃_{ξ} \end{matrix}} ( ∑_{involutions w_{ξ} ∈ Z_{S_{|ν_{ξ}|}} (ν_{ξ}), fixing no odd cycles of ν_{ξ} if d̃_{ξ} = −1}

(−1) \begin{matrix}|\{even cycles of ν_{ξ} fixed by w_{ξ} pointwise\}| \\ +|\{cycles of ν_{ξ} of length ≡ 2 mod 4 fixed by w_{ξ}, not pointwise\}| \\ + \frac{1}{2} |\{odd cycles of ν_{ξ} moved by w_{ξ}\}| \end{matrix}
× 2^{|{cycles of ν_{ξ} fixed by w_{ξ} pointwise}|})

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ 2|m̃_{ξ} \end{matrix}} ( ∑_{involutions w_{ξ} ∈ Z_{S_{|ν_{ξ}|}} (ν_{ξ}) fixing no odd cycles of ν_{ξ} if d̃_{ξ} = −1} (−2)^{|\{cycles of ν_{ξ} fixed by w_{ξ} pointwise\}|}) ,

and

⟨B\underline{ν}, Ind_{(K^{∘})^{F}}^{G^{F}} (1)⟩ = ⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ + ⟨B_{ζ̃.\underline{ν}}, Ind_{K^{F}}^{G^{F}} (1)⟩.

So in addition to (4.1.2) and (4.1.3), the combinatorial facts we need are:

∑_{involutions w ∈ Z_{S_{|ν|}} (ν)} (−1) \begin{matrix}|\{even cycles of ν fixed by w pointwise\}| \\ +|\{cycles of ν of length ≡ 2 mod 4 fixed by w, not pointwise\}| \\ + \frac{1}{2} |\{odd cycles of ν moved by w\}| \end{matrix}
× 2^{|{cycles of ν fixed by w pointwise}|}

= ∑_{\begin{matrix}λ⊢|ν| \\ 2|m_{2i}(λ) \end{matrix}}(−1)^{n(λ′)}(∏_{i}(m_{2i+1}(λ) + 1))χ_{ν}^{λ},

(4.3.2) and

∑_{involutions w ∈ Z_{S_{|ν|}} (ν) fixing no odd cycles of ν} (−1) \begin{matrix}|\{cycles of ν fixed by w pointwise\}| \\ +|\{cycles of ν of length ≡ 2 mod 4 fixed by w, not pointwise\}| \\ + \frac{1}{2} |\{odd cycles of ν moved by w\}| \end{matrix}

× 2^{|{cycles of ν fixed by w pointwise}|}

= ∑_{\begin{matrix}λ⊢|ν| \\ 2|m_{2i+1}(λ) \end{matrix}}(−1)^{n(λ′)}(∏_{i}(m_{2i}(λ) + 1))χ_{ν}^{λ}.

(4.3.3)

These will be proved in §5.

4.4. The U_{n}(𝔽_{q^{2}})/O_{n}^{±}(𝔽_{q}) and U_{n}(𝔽_{q^{2}})/SO_{n}^{±}(𝔽_{q}) Cases (n even). Finally, we suppose that n is even and F : G → G is non-split. So θF is induced by F_{V} as in §4.2, and we have the same dichotomy as to the Witt index of ⟨·, ·⟩ on V^{F_{V}}. Define ϵ ∈ {±1} as in §4.2, so that G^{F} ≅ U_{n}(𝔽_{q^{2}}), K^{F} ≅ O_{n}^{ϵ}(𝔽_{q}), (K^{∘})^{F} ≅ SO_{n}^{ϵ}(𝔽_{q}).

Theorem 4.4.1. For any \underline{ρ} ∈ P̂σ̃ ; n,

⟨χ\underline{ρ}, Ind_{O_{n}^{ϵ}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = \{ \frac{1}{2} ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=1 \\ 2∤m̃_{ξ} \end{matrix}} (∏_{i}(m_{2i}(ρ_{ξ}) + 1)) ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=−1 \\ 2∤m̃_{ξ} \end{matrix}}(∏_{i}(m_{2i+1}(ρ_{ξ}) + 1)) × ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=1 \\ 2|m̃_{ξ} \end{matrix}}(∏_{i}(m_{i}(ρ_{ξ}) + 1)), if d̃_{ξ} = 1, 2 ∤ m̃_{ξ} ⇒ 2|m_{2i+1}(ρ_{ξ}), ∀i, d̃_{ξ} = −1, 2 ∤ m̃_{ξ} ⇒ 2|m_{2i}(ρ_{ξ}), ∀i, and d̃_{ξ} = −1, 2 | m̃_{ξ} ⇒ ρ′_{ξ} is even 0, otherwise

+ \{ \begin{matrix}\frac{1}{2}ϵ, & if all ρ′_{ξ} are even \\ 0, & otherwise, and \end{matrix}
⟨χ\underline{ρ}, Ind_{SO_{n}^{ϵ}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ = ⟨χ\underline{ρ}, Ind_{O_{n}^{ϵ}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩ + ⟨χ^{ζ̃.\underline{ρ}}, Ind_{O_{n}^{ϵ}(𝔽_{q})}^{U_{n}(𝔽_{q^{2}} )}(1)⟩.

By (1.3.2), it is equivalent to say that for any \underline{ν} ∈ P̂σ̃ ; n,

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ = \frac{1}{2}(−1)^{\frac{n}{2}} ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=1 \\ 2∤m̃_{ξ} \end{matrix}} ( ∑_{\begin{matrix}ρ_{ξ}⊢|ν_{ξ}| \\ 2|m_{2i+1}(ρ_{ξ}) \end{matrix}}(−1)^{n(ρ′_{ξ})}(∏_{i}(m_{2i}(ρ_{ξ}) + 1))χ_{ν_{ξ}}^{ρ_{ξ}} )

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=−1 \\ 2∤m̃_{ξ} \end{matrix}} ( ∑_{\begin{matrix}ρ_{ξ}⊢|ν_{ξ}| \\ 2|m_{2i}(ρ_{ξ}) \end{matrix}}(−1)^{n(ρ′_{ξ})}(∏_{i}(m_{2i+1}(ρ_{ξ}) + 1))χ_{ν_{ξ}}^{ρ_{ξ}} )

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=1 \\ 2|m̃_{ξ} \end{matrix}}(−1)^{|ν_{ξ}|} ( ∑_{ρ_{ξ}⊢|ν_{ξ}|}(∏_{i}(m_{i}(ρ_{ξ}) + 1))χ_{ν_{ξ}}^{ρ_{ξ}} )

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ d̃_{ξ}=−1 \\ 2|m̃_{ξ} \end{matrix}} ( ∑_{\begin{matrix}ρ_{ξ}⊢|ν_{ξ}| \\ ρ′_{ξ} even \end{matrix}} χ_{ν_{ξ}}^{ρ_{ξ}} )

+ \frac{1}{2}ϵ ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ 2∤m̃_{ξ} \end{matrix}}(−1)^{\frac{|ν_{ξ}|}{2}} ( ∑_{\begin{matrix}ρ_{ξ}⊢|ν_{ξ}| \\ ρ′_{ξ} even \end{matrix}} χ_{ν_{ξ}}^{ρ_{ξ}} )

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ 2|m̃_{ξ} \end{matrix}} ( ∑_{\begin{matrix}ρ_{ξ}⊢|ν_{ξ}| \\ ρ′_{ξ} even \end{matrix}} χ_{ν_{ξ}}^{ρ_{ξ}} ) , and

⟨B\underline{ν}, Ind_{(K^{∘})^{F}}^{G^{F}} (1)⟩ = ⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ + ⟨B_{ζ̃.\underline{ν}}, Ind_{K^{F}}^{G^{F}} (1)⟩.

(4.4.1)

Here the sign

(−1)^{\frac{n}{2} +∑_{ξ∈⟨σ̃⟩\L} m̃_{ξ}n(ρ′_{ξ})+|ρ_{ξ}|} = (−1)^{\frac{n}{2} +∑_{ξ∈⟨σ̃⟩\L} m̃_{ξ}(n(ρ′_{ξ})+|ρ_{ξ}|)+|ρ_{ξ}|}

of (1.3.2) has been distributed in an obvious way (in the second term it has been

rewritten as ∏_{ξ∈⟨σ̃⟩\L}(−1)^{m̃_{ξ} ^{\frac{|ν_{ξ}|}{2}}} since |ρ_{ξ}| and n(ρ′_{ξ}) are even if ρ′_{ξ} is even).

The proof of this is similar to that of (4.2.1). Let (T, λ), {L_{(ξ,j,i)}} and w_{F} be as in §2.2. Lemma 4.3.2 again holds, with its usual proof. Corollary 4.2.3 must be modified as follows:

Corollary 4.4.2. The map f ↦ w_{f} induces a map

T^{F} \Θ_{T}^{F} /K^{F} → {involutions in Z_{W (T)}(w_{F})}.

If sign(w_{F}) = (−1)^{\frac{n}{2}} ϵ, this map is surjective. If sign(w_{F}) = −(−1)^{\frac{n}{2}} ϵ, the image consists of all involutions with at least one fixed point.

Proof. The proof is mostly the same as that of Corollary 4.2.3. Note that if f ∈ Θ_{T}^{F},

F_{V} (f^{−1}L_{(ξ,j,i)}) = θ(f)^{−1} ⋂_{\begin{matrix}(ξ′,j′,i′) \\ ≠w_{F} (ξ,j,i) \end{matrix}} L_{(ξ′,j′,i′)}^{⊥}

= ( ⨁_{\begin{matrix}(ξ′,j′,i′) \\ ≠w_{F} (ξ,j,i) \end{matrix}} f^{−1}L_{(ξ′,j′,i′)})^{⊥}

= f^{−1}L_{w_{f}^{−1}w_{F} (ξ,j,i)}.

So in (2) of the proof of Corollary 4.2.3, w_{F} should be replaced by w^{−1}w_{F}. Since sign(w) = (−1)^{\frac{n}{2}} if w is a fixed-point free involution, we get the result. □

For w an involution in Z_{W (T)}(w_{F}), define w̄ and i(w, ξ, j) as in the previous subsection.

Lemma 4.4.3. For f ∈ Θ_{T}^{F}, the number of T^{F} –K^{F} double cosets in T fK is

1,

if w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = 0 never occurs

2^{|{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0}|−1}, otherwise,

and the number of T^{F} –(K^{∘})^{F} double cosets in T fK^{∘} is

2^{|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0\}|},  \begin{matrix}if w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = 0 \\ ⇒ 2 | m_{ξ}(ν_{ξ})_{j} \end{matrix}
2^{|{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0}|−1}, otherwise.

Proof. As with Lemma 4.2.4, the proof of Lemma 4.1.4 applies again here. □

Now define Z′ and Z′′ as in §4.3. We now encounter the first major point of difference from §4.2:

Lemma 4.4.4. For f ∈ Θ_{T}^{F}, f ∈ Θ_{T,K,λ}^{F} ⇔ w_{f} ∈ Z′′, and f ∈ Θ_{T,K^{∘},λ}^{F} ⇔ w_{f} ∈ Z′ ∪ Z′′.

Proof. The descriptions of (T ∩ fKf^{−1})^{F} and λ|_{(T ∩fKf^{−1})^{F}} in the proof of Lemma 4.3.5 remain valid. The new feature when n is even is that

∑_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \end{matrix}} m̃_{ξ}(ν_{ξ})_{j}

is even, so that if t ∈ (T ∩ fKf^{−1})^{F} is such that

∑_{\begin{matrix}(ξ,j) \\ w̄_{f} (ξ,j)=(ξ,j) \\ i(w_{f} ,ξ,j)=0 \\ α_{(ξ,j)}=−1 \end{matrix}} m̃_{ξ}(ν_{ξ})_{j}

is odd, then

𝔽_{q}-rank(Z_{G}((T ∩ fKf^{−1})^{∘})) − 𝔽_{q}-rank(Z_{G}^{∘}(t) ∩ Z_{G}((T ∩ fKf^{−1})^{∘})) = 1,

K^{F} -INVARIANTS IN IRREDUCIBLE REPRESENTATIONS OF G^{F}, WHEN G = GL_{n} 41

so ϵ_{T,f}(t) = −1. In fact, ϵ_{T,f} is the function

\begin{matrix}((α_{(ξ,j)}), (β_{(ξ,j)}), (γ_{(ξ,j)})) & ↦ \\ (−1)^{|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0,2∤m̃_{ξ},2∤(ν_{ξ})_{j},α_{(ξ,j)}=−1\}|}. \end{matrix}

Hence f ∈ Θ_{T,K,λ}^{F} if and only if for all (ξ, j) such that w̄_{f}(ξ, j) = (ξ, j) and i(w_{f}, ξ, j) = 0,

d̃_{ξ} = −1, 2 ∤ (ν_{ξ})_{j} ⇔ 2 ∤ m̃_{ξ}, 2 ∤ (ν_{ξ})_{j},

besides the usual conditions on other (ξ, j). Clearly this is equivalent to w_{f} ∈ Z′′. The second statement is proved similarly. □

As in §4.3,

𝔽_{q}-rank(T)+𝔽_{q}-rank(Z_{G}((T ∩ fKf^{−1})^{∘}))

≡ \frac{n}{2} + |\{(ξ, j) | w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = 0, 2|m̃_{ξ}(ν_{ξ})_{j}\}|

+ |\{(ξ, j) | w̄_{f}(ξ, j) = (ξ, j), i(w_{f}, ξ, j) = \frac{1}{2}m̃_{ξ}(ν_{ξ})_{j}, 4 ∤ m̃_{ξ}(ν_{ξ})_{j}\}|

+ \frac{1}{2}|\{(ξ, j) | w̄_{f}(ξ, j) ≠ (ξ, j), 2 ∤ m̃_{ξ}(ν_{ξ})_{j}\}| mod 2.

So Lusztig’s formula gives (compare §4.2):

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩

= \frac{1}{2}(−1)^{\frac{n}{2}} ∑_{w∈Z′′}(−1) \begin{matrix}|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0,2|m̃_{ξ}(ν_{ξ})_{j}\}| \\ +|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j},4∤m̃_{ξ}(ν_{ξ})_{j}\}| \\ + \frac{1}{2} |\{(ξ,j) | w̄_{f} (ξ,j)≠(ξ,j),2∤m̃_{ξ}(ν_{ξ})_{j}\}| \end{matrix}

× 2^{|{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0}|}

+ \frac{1}{2}ϵ ∑_{w∈Z′′′}(−1) \begin{matrix}|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j},4|m̃_{ξ}(ν_{ξ})_{j}\}| \\ + \frac{1}{2} |\{(ξ,j) | w̄_{f} (ξ,j)≠(ξ,j),2∤m̃_{ξ}(ν_{ξ})_{j}\}| \end{matrix} , and

⟨B\underline{ν}, Ind_{(K^{∘})^{F}}^{G^{F}} (1)⟩

= \frac{1}{2}(−1)^{\frac{n}{2}} ∑_{w∈Z′}(−1) \begin{matrix}|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0,2|m̃_{ξ}(ν_{ξ})_{j}\}| \\ +|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j},4∤m̃_{ξ}(ν_{ξ})_{j}\}| \\ + \frac{1}{2} |\{(ξ,j) | w̄_{f} (ξ,j)≠(ξ,j),2∤m̃_{ξ}(ν_{ξ})_{j}\}| \end{matrix}

× 2^{|{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0}|}

+ \frac{1}{2}(−1)^{\frac{n}{2}} ∑_{w∈Z′′}(−1) \begin{matrix}|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)=0,2|m̃_{ξ}(ν_{ξ})_{j}\}| \\ +|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j},4∤m̃_{ξ}(ν_{ξ})_{j}\}| \\ + \frac{1}{2} |\{(ξ,j) | w̄_{f} (ξ,j)≠(ξ,j),2∤m̃_{ξ}(ν_{ξ})_{j}\}| \end{matrix}

× 2^{|{(ξ,j) | w̄(ξ,j)=(ξ,j),i(w,ξ,j)=0}|}

+ ϵ ∑_{w∈Z′′′}(−1) \begin{matrix}|\{(ξ,j) | w̄_{f} (ξ,j)=(ξ,j),i(w_{f} ,ξ,j)= \frac{1}{2} m̃_{ξ}(ν_{ξ})_{j},4|m̃_{ξ}(ν_{ξ})_{j}\}| \\ + \frac{1}{2} |\{(ξ,j) | w̄_{f} (ξ,j)≠(ξ,j),2∤m̃_{ξ}(ν_{ξ})_{j}\}| \end{matrix} .

42 ANTHONY HENDERSON

As in §4.1, we transform these expressions to get

⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩

= \frac{1}{2}(−1)^{\frac{n}{2}}

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ 2∤m̃_{ξ} \end{matrix}} ( ∑_{involutions w_{ξ} ∈ Z_{S_{|ν_{ξ}|}} (ν_{ξ}), fixing no odd cycles of ν_{ξ} if d̃_{ξ} = 1}

(−1) \begin{matrix}|\{cycles of ν_{ξ} fixed by w_{ξ} pointwise\}| \\ +|\{cycles of ν_{ξ} of length ≡ 2 mod 4 fixed by w_{ξ}, not pointwise\}| \\ + \frac{1}{2} |\{odd cycles of ν_{ξ} moved by w_{ξ}\}| \end{matrix}
× 2^{|{cycles of ν_{ξ} fixed by w_{ξ} pointwise}|})

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ 2|m̃_{ξ} \end{matrix}} ( ∑_{involutions w_{ξ} ∈ Z_{S_{|ν_{ξ}|}} (ν_{ξ}), fixing no odd cycles of ν_{ξ} if d̃_{ξ} = −1} (−2)^{|\{cycles of ν_{ξ} fixed by w_{ξ} pointwise\}|})

+ \frac{1}{2}ϵ ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ 2∤m̃_{ξ} \end{matrix}} ( ∑_{fixed-point free involutions w_{ξ} ∈ Z_{S_{|ν_{ξ}|}} (ν_{ξ})} (−1) \begin{matrix}|\{cycles of ν_{ξ} divisible by 4, fixed by w_{ξ}\}| \\ + \frac{1}{2} |\{odd cycles of ν_{ξ}\}| \end{matrix})

× ∏_{\begin{matrix}ξ∈⟨σ̃⟩\L \\ 2|m̃_{ξ} \end{matrix}} ( ∑_{fixed-point free involutions w_{ξ} ∈ Z_{S_{|ν_{ξ}|}} (ν_{ξ})}(−1)^{|\{cycles of ν_{ξ} fixed by w_{ξ}\}|}) ,

and

⟨B\underline{ν}, Ind_{(K^{∘})^{F}}^{G^{F}} (1)⟩ = ⟨B\underline{ν}, Ind_{K^{F}}^{G^{F}} (1)⟩ + ⟨B_{ζ̃.\underline{ν}}, Ind_{K^{F}}^{G^{F}} (1)⟩.

So (4.4.1) follows by applying (4.3.2), (4.3.3), (4.1.2), (4.1.3), (3.3.3) and (3.1.3).

5. Combinatorics of the Symmetric Group

This section is devoted to the proof of the combinatorial facts invoked in §§2-4. We say that a function f on the set of partitions is multiplicative if

f(ν) = ∏_{i} f(i^{m_{i}(ν)}), ∀ν.

Examples of multiplicative functions of ν are the sign ϵ_{ν}, z_{ν} = |Z_{S_{|ν|}}(ν)|, and |{involutions in Z_{S_{|ν|}}(ν)}|.

Our first starting point is [14, VII.(2.4)], which can be rewritten:

|\{fixed-point free involutions in Z_{S_{|ν|}}(ν)\}| = ∑_{\begin{matrix}λ⊢|ν| \\ λ even \end{matrix}} χ_{ν}^{λ}.

This is precisely (2.1.2). Now if there exists an involution w in Z_{S_{|ν|}}(ν), then |{cycles of ν moved by w}| is even, so

(−1)^{|{cycles of ν fixed by w}|} = (−1)^{|{cycles of ν}|}.

As already observed above, if there exists a fixed-point free involution w in Z_{S_{|ν|}}(ν), then

(−1)^{|{cycles of ν}|} = (−1)^{|{even cycles of ν}|} = ϵ_{ν}.

Thus

∑_{fixed-point free involutions w ∈ Z_{S_{|ν|}} (ν)} (−1)^{|\{cycles of ν fixed by w\}|} = ϵ_{ν} ∑_{\begin{matrix}λ⊢|ν| \\ λ even \end{matrix}} χ_{ν}^{λ} = ∑_{\begin{matrix}λ⊢|ν| \\ λ′ even \end{matrix}} χ_{ν}^{λ},

which proves (3.1.3). Now (4.1.3) follows from (2.1.2) and the fact that

(−1)^{|{even cycles of ν}|} ∑_{involutions w ∈ Z_{S_{|ν|}} (ν) fixing no odd cycles of ν}

(−2)^{|{cycles of ν fixed by w pointwise}|}

= |\{ \begin{matrix}fixed-point free \\ involutions in Z_{S_{|ν|}}(ν) \end{matrix} \}| .

To see this, note that since both sides are multiplicative, it suffices to consider the case when ν is of the form (a^{b}), in which case both sides are

0,  if a is odd and b is odd,
a^{b/2} \frac{b!}{2^{b/2}(\frac{b}{2})!},  if a is odd and b is even, and

∑_{r=0}^{⌊ \frac{b}{2} ⌋} \binom{b}{2r}a^{r} \frac{(2r)!}{2^{r}r!}, if a is even.

This fact also implies (3.2.2), as clearly

∑_{involutions w ∈ Z_{S_{|ν|}} (ν) with signed fixed points, signs reversed by ν} (−1)^{|{cycles of ν fixed by w, not pointwise}|}

= ∑_{involutions w ∈ Z_{S_{|ν|}} (ν) fixing no odd cycles of ν} (−1)^{|{cycles of ν fixed by w, not pointwise}|}

× 2^{|{cycles of ν fixed by w pointwise}|}

= (−1)^{|{even cycles of ν}|} ∑_{involutions w ∈ Z_{S_{|ν|}} (ν) fixing no odd cycles of ν} (−2)^{|{cycles of ν fixed by w pointwise}|},

since if there exists an involution w ∈ Z_{S_{|ν|}}(ν) fixing no odd cycles of ν, then |{odd cycles of ν}| must be even. Similarly, (3.3.3) follows from (2.1.2) and the fact that

∑_{fixed-point free involutions w ∈ Z_{S_{|ν|}} (ν)} (−1)^{|\{cycles of ν divisible by 4, fixed by w\}|+ \frac{1}{2} |\{odd cycles of ν\}|}

= (−1)^{\frac{|ν|}{2} +|\{even cycles of ν\}|} |\{ \begin{matrix}fixed-point free \\ involutions in Z_{S_{|ν|}}(ν) \end{matrix} \}| .

Again both sides are multiplicative, so it suffices to prove this when ν = (a^{b}), in which case it is easy.

Our second starting point is [14, I.8 Example 11], which can be rewritten:

∑_{involutions w ∈ Z_{S_{|ν|}} (ν)} (−1)^{|{cycles of ν fixed by w, not pointwise}|} = ∑_{λ⊢|ν|} χ_{ν}^{λ}.

Now (3.3.4) follows from this and the fact that

|\{ \begin{matrix}involutions w ∈ Z_{S_{|ν|}}(ν) \\ fixing no even cycles of ν \end{matrix} \}|

= ∑_{involutions w ∈ Z_{S_{|ν|}} (ν)} (−1)^{|{cycles of ν fixed by w, not pointwise}|},

which we can again prove simply by observing that when ν = (a^{b}) both sides are

0,  if a is even and b is odd,
a^{b/2} \frac{b!}{2^{b/2}(\frac{b}{2})!},  if a is even and b is even, and

∑_{r=0}^{⌊ \frac{b}{2} ⌋} \binom{b}{2r}a^{r} \frac{(2r)!}{2^{r}r!}, if a is odd.

Furthermore (3.1.4) follows from (3.3.4), since if there exists an involution w ∈ Z_{S_{|ν|}}(ν) fixing no even cycles of ν, then |{even cycles of ν}| is even, so

(−1)^{|{cycles of ν fixed by w}|} = (−1)^{|{odd cycles of ν}|}
= (−1)^{|ν|}.

The remaining identities require a different approach. The prototype is (3.1.2). By definition of induction product ([14, I.7]), and using (3.1.3),

∑_{involutions w ∈ Z_{S|ν|} (ν) with signed fixed points, signs stable under ν, signature (p^{+}, p^{−}) + ℤ}

(−1)^{|{cycles of ν fixed by w, not pointwise}|}

= ∑_{r=⌈ \frac{p^{+}+p^{−}−|ν|}{2} ⌉}^{min\{p^{+},p^{−}\}} ((  ∑_{\begin{matrix}μ⊢|ν|+2r−p^{+}−p^{−} \\ μ′ even \end{matrix}} χ^{μ}).χ^{(p^{+}−r)}.χ^{(p^{−}−r)}) (ν)

= ∑_{r=⌈ \frac{p^{+}+p^{−}−|ν|}{2} ⌉}^{min\{p^{+},p^{−}\}} ∑_{λ⊢|ν|} b(p^{+} − r, p^{−} − r, λ′)χ_{ν}^{λ},

where, by Pieri’s formula ([14, (5.16)]), b(p^{+} − r, p^{−} − r, λ′) is the number of ways of removing first a vertical (p^{−} − r)-strip, then a vertical (p^{+} − r)-strip, from the Young diagram of λ′, to leave a diagram with all rows of even length. Now every signed tableaux of shape λ′ and signature (p^{+}, p^{−}) + ℤ determines uniquely an r as above and such a way of removing strips, as follows:

• order rows of equal length so that rows ending ⊟ are below rows ending ⊞;

• take the vertical (p^{−} − r)-strip to consist of all final boxes signed ⊟;

• take the vertical (p^{+} − r)-strip to consist of all final boxes signed ⊞ in rows of odd length, including those made odd by removal of the first strip.

This correspondence is clearly bijective, which proves (3.1.2).

The proof of (3.3.2) is similar: using (3.3.3),

∑_{involutions w ∈ Z_{S_{|ν|}} (ν) with signed fixed points, signs stable under ν, signature (p^{+}, p^{−}) + ℤ} (−1) \begin{matrix}|\{cycles of ν divisible by 4, fixed by w, not pointwise\}| \\ + \frac{1}{2} |\{odd cycles of ν moved by w\}| \end{matrix}

= ∑_{r=⌈ \frac{p^{+}+p^{−}−|ν|}{2} ⌉}^{min\{p^{+},p^{−}\}} (((−1)^{\frac{|ν|+2r−p^{+}−p^{−}}{2}} ∑_{\begin{matrix}μ⊢|ν|+2r−p^{+}−p^{−} \\ μ′ even \end{matrix}} χ^{μ}).χ^{(p^{+}−r)}.χ^{(p^{−}−r)}) (ν)

= ∑_{λ⊢|ν|} c(p^{+}, p^{−}, λ′)χ_{ν}^{λ},

where by the same bijection as before,

c(p^{+}, p^{−}, λ′)

= ∑_{signed tableaux T of shape λ′, signature (p^{+}, p^{−}) + ℤ} (−1) \begin{matrix}\frac{1}{2} (|λ|−|\{rows of T ending ⊟\}| \\ −|\{odd rows of T ending ⊞\}|−|\{even rows of T ending ⊟\}|) \end{matrix}

= (−1)^{\frac{1}{2} (|λ|−|\{odd parts of λ′\}|)} ∑_{signed tableaux T of shape λ′, signature (p^{+}, p^{−}) + ℤ} (−1)^{|\{even rows of T ending ⊟\}|}.

Now

(−1)^{\frac{1}{2} (|λ|−|\{odd parts of λ′\}|)} = (−1)^{|\{parts of λ′ ≡ 2 or 3 mod 4\}|}
= (−1)∑^{i} ^{\binom{λ′_{i}}{2}} = (−1)^{n(λ)},

and by grouping together signed tableaux which differ only in even rows it is easy to see that the sum equals ā(p^{+}, p^{−}, λ′). So (3.3.2) is proved.

Our next task is to modify this proof of (3.3.2) to derive (3.4.2). (The reader is warned that the following argument is not very satisfactory, and implored to find a better one.) We may assume that |ν| is even, for otherwise both sides vanish. Following the above pattern, we need to replace χ^{(p^{+}−r)}.χ^{(p^{−}−r)} with the class function on S_{|ν|−2r} defined by

w ↦ |{(A^{+}, A^{−}) | {1, ⋯, |ν| − 2r} = A^{+}∐A^{−}, |A^{+}| = |A^{−}|, w(A^{±}) = A^{∓}}|.

We claim that this class function is

∑_{i=0}^{\frac{|ν|}{2} −r}(−1)^{i}χ^{(|ν|−2r−i,i)}.

To see this, note that the class function

w ↦ |{(A^{+}, A^{−}) | {1, ⋯, |ν| − 2r} = A^{+}∐A^{−}, |A^{+}| = |A^{−}|, w(A^{±}) = A^{±}}|

is

Ind_{S_{\frac{|ν|}{2} −r}×S_{\frac{|ν|}{2} −r}}^{S_{|ν|−2r}}(1) = χ^{( \frac{|ν|}{2} −r)}.χ^{( \frac{|ν|}{2} −r)} = ∑_{i=0}^{\frac{|ν|}{2} −r} χ^{(|ν|−2r−i,i)}

by Pieri’s formula. So an irreducible representation of S_{|ν|−2r} has nonzero fixed vectors for S_{\frac{|ν|}{2} −r} × S_{\frac{|ν|}{2} −r} if and only if its character is χ^{(|ν|−2r−i,i)} for some i, and if so the space of fixed vectors is one-dimensional. It is easy to check (for instance using explicit models for these representations) that these fixed vectors are also fixed by the larger group (S_{\frac{|ν|}{2} −r} × S_{\frac{|ν|}{2} −r}) ⋊ S_{2} (where S_{2} permutes the factors) if and only if i is even. Hence

Ind_{(S_{\frac{|ν|}{2} −r}×S_{\frac{|ν|}{2} −r})⋊S_{2}}^{S_{|ν|−2r}}(1) = ∑_{\begin{matrix}i=0 \\ i even \end{matrix}}^{\frac{|ν|}{2} −r} χ^{(|ν|−2r−i,i)}.

As a class function on S_{|ν|−2r}, this is

w ↦ |{{A^{+}, A^{−}} | {1, ⋯, |ν| − 2r} = A^{+}∐A^{−},
|A^{+}| = |A^{−}|, w{A^{+}, A^{−}} = {A^{+}, A^{−}}}|

= \frac{1}{2}|\{(A^{+}, A^{−}) | \{1, ⋯, |ν| − 2r\} = A^{+}∐A^{−}, |A^{+}| = |A^{−}|, w(A^{±}) = A^{±}\}|

+ \frac{1}{2}|\{(A^{+}, A^{−}) | \{1, ⋯, |ν| − 2r\} = A^{+}∐A^{−}, |A^{+}| = |A^{−}|, w(A^{±}) = A^{∓}\}|,

so our claim is proved. Thus

∑_{involutions w ∈ Z_{S_{|ν|}} (ν) with signed fixed points, signs reversed under ν} (−1) \begin{matrix}|\{cycles of ν divisible by 4, fixed by w, not pointwise\}| \\ + \frac{1}{2} |\{odd cycles of ν moved by w\}| \end{matrix}

= ∑_{r=0}^{\frac{|ν|}{2}} (((−1)^{r} ∑_{\begin{matrix}μ⊢2r \\ μ′ even \end{matrix}} χ^{μ}).( ∑_{i=0}^{\frac{|ν|}{2} −r}(−1)^{i}χ^{(|ν|−2r−i,i)})) (ν)

= ∑_{λ⊢|ν|} d(λ′)χ_{ν}^{λ},

where, analogously to the above, we can write d(λ′) as the sum, over signed tableaux of shape λ′ and signature (\frac{|λ|}{2}, \frac{|λ|}{2}), of a sign determined by T. As in the proof of (3.3.2), the (−1)^{r} contribution to the sign is

(−1)^{n(λ)+|{even rows of T ending ⊟}|}.

The (−1)^{i} contribution is trickier to rephrase in terms of T, but an examination of the bijection

Tab(λ′ − μ′, (\frac{|ν|}{2}, \frac{|ν|}{2})) →^{∼} ∐_{i=0}^{\frac{|ν|}{2} −r} Tab^{0}(λ′ − μ′, (|ν| − 2r − i, i))

defined in [14, (9.4)] (in the proof of the Littlewood-Richardson Rule) reveals that the correct reformulation is

(−1)^{|{rows of T ending ⊟}|−m(T)},

where m(T) is the maximum, over all rows R of T, of the quantity

|{odd rows ending ⊟ below or equal to R}|

− |{odd rows ending ⊞ below or equal to R}|

(assuming that the rows of T are ordered, as above, so that rows ending ⊟ come below rows ending ⊞ of the same length). Hence

d(λ′) = (−1)^{n(λ)} ∑_{signed tableaux T of shape λ′, signature ( \frac{|λ|}{2} , \frac{|λ|}{2} )} (−1)^{|\{odd rows of T ending ⊟\}|−m(T)},

and we are reduced to proving that

∑_{signed tableaux T of shape λ′, signature ( \frac{|λ|}{2} , \frac{|λ|}{2} )}

(−1)^{|{odd rows of T ending ⊟}|−m(T)}

= \{ \begin{matrix}∏_{i}(m_{2i}(λ′) + 1), & if 2 | m_{2i+1}(λ′), ∀i \\ 0, & otherwise. \end{matrix}

Grouping together signed tableaux which differ only in even rows, we see that we may assume that λ′ has only odd parts. In lieu of a direct proof, we can deduce this from (3.2.2), proved above. It says that

∑_{\begin{matrix}λ⊢|ν| \\ λ even \end{matrix}} χ_{ν}^{λ} = ∑_{involutions w ∈ Z_{S_{|ν|}} (ν) with signed fixed points, signs reversed under ν} (−1)^{|\{cycles of ν fixed by w, not pointwise\}|}

= ∑_{r=0}^{\frac{|ν|}{2}} (( ∑_{\begin{matrix}μ⊢2r \\ μ′ even \end{matrix}} χ^{μ}).( ∑_{i=0}^{\frac{|ν|}{2} −r}(−1)^{i}χ^{(|ν|−2r−i,i)})) (ν),

whence

∑_{signed tableaux T of shape λ′, signature ( \frac{|λ|}{2} , \frac{|λ|}{2} )} (−1)^{|\{rows of T ending ⊟\}|−m(T)} = \{ \begin{matrix}1, & if 2 | m_{i}(λ′), ∀i \\ 0, & otherwise. \end{matrix}

When λ′ has only odd parts, this is precisely the statement we want.

48 ANTHONY HENDERSON

It is now easy to prove (4.3.3), since ∑_{involutions w ∈ Z_{S_{|ν|}} (ν) fixing no odd cycles of ν} (−1) \begin{matrix}|\{cycles of ν fixed by w pointwise\}| \\ +|\{cycles of ν of length ≡ 2 mod 4 fixed by w, not pointwise\}| \\ + \frac{1}{2} |\{odd cycles of ν moved by w\}| \end{matrix}

× 2^{|{cycles of ν fixed by w pointwise}|}

= (−1)^{|\{cycles of ν\}|} ∑_{involutions w ∈ Z_{S_{|ν|}} (ν) with signed fixed points, signs reversed under ν} (−1) \begin{matrix}|\{cycles of ν divisible by 4, fixed by w, not pointwise\}| \\ + \frac{1}{2} |\{odd cycles of ν moved by w\}| \end{matrix}

= ϵ_{ν} ∑_{\begin{matrix}λ⊢|ν| \\ 2|m_{2i+1}(λ′) \end{matrix}}(−1)^{n(λ)}(∏_{i}(m_{2i}(λ′) + 1))χ_{ν}^{λ}

= ∑_{\begin{matrix}λ⊢|ν| \\ 2|m_{2i+1}(λ) \end{matrix}}(−1)^{n(λ′)}(∏_{i}(m_{2i}(λ) + 1))χ_{ν}^{λ},

as required. Finally, (4.1.2) follows from (3.1.2) by summing over all signatures: ∑_{involutions w ∈ Z_{S_{|ν|}} (ν)} (−2)^{|{cycles of ν fixed by w pointwise}|} = (−1)^{|{cycles of ν}|} ∑_{involutions w ∈ Z_{S_{|ν|}} (ν) with signed fixed points, signs stable under ν} (−1)^{|{cycles of ν fixed by w, not pointwise}|} = (−1)^{|ν|}ϵ_{ν} ∑_{λ⊢|ν|} |{signed tableaux of shape λ′}|χ_{ν}^{λ} = (−1)^{|ν|} ∑_{λ⊢|ν|} |{signed tableaux of shape λ}|χ_{ν}^{λ}

= (−1)^{|ν|} ∑_{λ⊢|ν|}(∏_{i}(m_{i}(λ) + 1))χ_{ν}^{λ},

and (4.3.2) follows from (3.3.2) in an analogous way:

∑_{involutions w ∈ Z_{S_{|ν|}} (ν)} (−1) \begin{matrix}|\{even cycles of ν fixed by w pointwise\}| \\ +|\{cycles of ν of length ≡ 2 mod 4 fixed by w, not pointwise\}| \\ + \frac{1}{2} |\{odd cycles of ν moved by w\}| \end{matrix}

× 2^{|{cycles of ν fixed by w pointwise}|}

= ϵ_{ν} ∑_{λ⊢|ν|}(−1)^{n(λ)} |\{ \begin{matrix}signed tableaux of shape λ′, \\ fixed under inverting all rows \end{matrix} \}| χ_{ν}^{λ}

= ∑_{\begin{matrix}λ⊢|ν| \\ 2|m_{2i}(λ) \end{matrix}}(−1)^{n(λ′)}(∏_{i}(m_{2i+1}(λ) + 1))χ_{ν}^{λ}.

References

[1] E. Bannai, N. Kawanaka, and S.-Y. Song, The character table of the Hecke algebra H(GL_{2n}(𝔽_{q}), Sp_{2n}(𝔽_{q})), J. Algebra, 129 (1990), pp. 320–366.

[2] D. Barbasch and D. A. Vogan jr, Weyl group representations and nilpotent orbits, in Proceedings of the Park City Conference on Harmonic Analysis, P. Trombi ed., Progress in Mathematics, Birkhauser (Boston), 1982, pp. 21–33.

[3] P. Deligne and G. Lusztig, Representations of reductive groups over a finite field, Ann. of Math., 103 (1976), pp. 103–161.

[4] J. A. Green, The characters of the finite general linear groups, Trans. Amer. Math. Soc., 80 (1955), pp. 402–447.

[5] A. Henderson, Spherical functions and character sheaves. Available from www.maths.usyd.edu.au:8000/u/anthonyh/.

[6] G. Lusztig, Coxeter orbits and eigenspaces of Frobenius, Invent. Math., 28 (1976), pp. 101– 159.

[7] , Characters of reductive groups over a finite field, in Proceedings of the International Congress of Mathematicians, Warszawa, 1983, pp. 877–880.

[8] , Left cells in Weyl groups, in Lie Groups Representations, no. 1024 in Lecture Notes in Math., 1983, pp. 99–111.

[9] , Characters of Reductive Groups over a Finite Field, vol. 107 of Ann. of Math. Studies, Princeton University Press, 1984.

[10] , On the representations of reductive groups with disconnected center, Astérisque, (1988), pp. 157–166.

[11] , Symmetric spaces over a finite field, in The Grothendieck Festschrift, III, no. 88 in Progress in Mathematics, Birkhauser (Boston), 1990, pp. 57–81.

[12] , G(𝔽_{q})-invariants in irreducible G(𝔽_{q^{2}})-modules, Represent. Theory, 4 (2000), pp. 446–465.

[13] G. Lusztig and B. Srinivasan, The characters of the finite unitary groups, J. Algebra, 49 (1977), pp. 167–171.

[14] I. G. Macdonald, Symmetric Functions and Hall Polynomials, Oxford Univ. Press, second ed., 1995.

[15] P. E. Trapa, Generalized Robinson-Schensted algorithms for real groups, Intern. Math. Res. Notices, 1999, no. 15, pp. 803–834.

[16] A. Yamamoto, Orbits in the flag variety and images of the moment map for classical groups I, Represent. Theory, 1 (1997), pp. 329–404.

School of Mathematics and Statistics, University of Sydney, NSW 2006, AUSTRALIA

E-mail address: anthonyh@maths.usyd.edu.au