I received the following answer from the Star-K but it doesn't include detailed sources:
The way this system works is that, when you open a faucet, hot water
exits the tank. Cold water enters the tank in order to replace the
hot water which has exited. The system recognizes the change in
temperature in the tank, and activates the tankless heater.
Presumably, it takes at least a few seconds for all this to happen.
Since there is a time delay, Rabbi Heinemann would consider the
activation of tankless heater to be a gramma, which is permissible on
Yom Tov. Therefore this should be fine to use on Yom Tov.
His only concern was that maybe the system is so efficient and the
thermostat so sensitive that there is not a time delay of a few
seconds before the tankless heater activates. He feels that this is
unlikely, especially if the system is not set at the hottest setting.
Perhaps you could check with the installer (or manufacturer) that
there is a delay of a few seconds between the opening of the faucet
and the activation of the tankless heater.
Basically, the ruling (according to the Star-K) is directly dependent upon the sensitivity of the system to fluctuations in temperature - if it turns on immediately (or perhaps even if it preheats the inflow of water?) when the tap is used, then it would be problematic.
This need for grama would seem to apply especially when the (gas-based) tankless water heater doesn't have a pilot light, since any time it activates you would be starting a new fire without an existing flame.
I have not received an answer to the "counterflow" question of hatmana of chamin bitzonen.