Groups without faithful transitive permutation representations of small degree

A subgroup H of a group G is core-free if H contains no non-trivial normal subgroup of G, or equivalently the transitive permutation representation of G on the cosets of H is faithful. We study the obstacles to a group having large core-free subgroups. We call a subgroup D a ''dedekind'' subgroup of G if all subgroups of D are normal in G. Our main result is the following: If a finite group G has no core-free subgroups of order greater than k, then G has two dedekind subgroups D-1 and D-2 such that every subgroup in G of order greater than f(k) has non-trivial intersection with either D-1 or D-2 (where f is a fixed function independent of G). Examples show that the dedekind subgroups need not have index bounded by a function of k, and the result would not be true with one dedekind subgroup instead of two. We exhibit various related properties of p-groups and infinite locally finite groups without large core-free subgroups, including the following: If G is a locally finite group with no infinite core-free subgroup, then every infinite subgroup of G contains a non-trivial cyclic normal subgroup of G. We also exhibit asymptotic bounds for some related problems, including the following: If a group G has a solvable subgroup of index n, then G has a solvable normal subgroup of index at most n(c) (for some absolute constant c). If G is a transitive permutation group of degree n with cyclic point-stabilizer subgroup, then \G\ < n(7). (C) 1997 Academic Press.