Sounds like a job for an LDO. You can find them with drop-out under a hundred mV. If output voltage has to run above battery voltage minus regulator drop-out, you'd have to use a small buck-boost regulator. There are buck-boost charge pumps that are easy to wire and have a small footprint, just a chip and three caps, input, output, and switch. For variable output voltage there would be a feedback divider to set voltage as well, just two resistors. The Microchip MCP1252 would be an example.