Optimization is one of the key applications of quantum computing where a quantum speedup has been an eagerly anticipated outcome. A promising approach to optimization using quantum dynamics is to consider a linear combination s(t)B + [1 - s(t)]C of two noncommuting Hamiltonians B and C, where C encodes the solution to the optimization problem in its ground state, B is a Hamiltonian whose ground state is easy to prepare, and s(t) is an element of [0, 1] is the bounded "switching schedule" or "path," with t is an element of [0, t(f)]. This approach encompasses two of the most widely studied quantum-optimization algorithms: quantum annealing [QA; continuous s(t)] and the quantum approximate optimization algorithm [QAOA; piecewise constant s(t)]. While it is notoriously difficult to prove a quantum advantage for either algorithm, it is possible to compare and contrast them by finding the optimal s(t). Here we provide a rigorous analysis of this quantum optimal control problem, entirely within the geometric framework of Pontryagin's maximum principle of optimal control theory. We extend earlier results, derived in a purely closed-system setting, to open systems. This is the natural setting for experimental realizations of QA and QAOA. In the closed-system setting it was shown that the optimal solution is a "bang-anneal-bang" schedule, with the bangs characterized by s(t) = 0 and s(t) = 1 in finite subintervals of [0, t(f) ], in particular, s(0) = 0 and s(t(f)) = 1, in contrast to the standard prescription s(0) = 1 and s(t(f)) = 0 of QA. As an example, we prove that for a single spin-1/2, the optimal solution in the closed-system setting is the bang-bang schedule, switching midway from s equivalent to 0 to s equivalent to 1. For finite-dimensional environments and without any approximations we identify sufficient conditions ensuring that either the bang-anneal, anneal-bang, or bang-anneal-bang schedules are optimal, and recover the optimality of s(0) = 0 and s(t(f)) = 1. However, for infinite-dimensional environments and a system described by an adiabatic Redfield master equation we do not recover the bang-type optimal solution. In fact we can only identify conditions under which s(t(f)) = 1, and even this result is not recovered in the fully Markovian limit, suggesting that the pure anneal-type schedule is optimal. Our open-system results have implications for the use of experimental quantum-information processors, which are by necessity noisy, and suggest that in this practical sense the optimal schedules for quantum optimization are likely to be continuous.