RADON TRANSFORM ON SYMMETRIC MATRIX DOMAINS

Let K = R, C, H be the field of real, complex or quaternionic numbers and M-p,M-q( K) the vector space of all p x q-matrices. Let X be the matrix unit ball in M-n-r,M-r(K) consisting of contractive matrices. As a symmetric space, X = G/K = O(n - r, r)/O( n - r) x O( r), U( n - r, r)/U( n - r) x U( r) and respectively Sp( n - r, r)/Sp( n - r) x Sp( r). The matrix unit ball y(0) in M-r'-r,M-r with r' <= n - 1 is a totally geodesic submanifold of X and let Y be the set of all G-translations of the submanifold y(0). The set Y is then a manifold and an a. ne symmetric space. We consider the Radon transform Rf(y) for functions f is an element of C-0(infinity) (X) defined by integration of f over the subset y, and the dual transform (RF)-F-t(x), x is an element of X for functions F(y) on Y. For 2r < n, 2r <= r' with a certain evenness condition in the case K = R, we find a G-invariant differential operator M and prove it is the right inverse of (RR)-R-t, R(t)RMf = cf, for f is an element of C-0(infinity) (X), c not equal 0. The operator f --> R(t)Rf is an integration of f against a ( singular) function determined by the root systems of X and y(0). We study the analytic continuation of the powers of the function and we find a Bernstein-Sato type formula generalizing earlier work of the author in the set up of the Berezin transform. When X is a rank one domain of hyperbolic balls in Kn-1 and y(0) is the hyperbolic ball in Kr'-1, 1 < r' < n we obtain an inversion formula for the Radon transform, namely MR(t)Rf = cf. This generalizes earlier results of Helgason for non-compact rank one symmetric spaces for the case r' <= n - 1.