Is it permitted for a patient with a terminal illness to accept a dangerous experimental surgery? (Every case is different and requires CYLOR, but I'm just curious about the two sides of the argument - especially if Rav Moshe writes on the topic)
SA Yoreh Deah 55:1 (transl: Nishmat Avraham)
One may not receive treatment from an idolater for any injury or disease which is so dangerous that Shabbat laws could be set aside for the purpose. However, if he is a recognized specialist, it is permitted. If he is not, one may not receive treatment from him even if there is a possibility that the disease may be fatal. However, if he would certainly die, he may go to him for treatment for, in such a case, there is no fear that one's life may be shortened by the treatment. [...]
Commenting on this, Nishmat Avraham writes (vol 2 pp. 55ff) that
the Gilyon Maharsha was asked regarding a patient who was told that if treated with certain drugs he would be cured but there was also the possibility that he would die immediately. On the other hand, if left untreated he would certainly die later. He wrote that the patient mat take the treatment based on the above Shulchan Aruch.
Nshmat Avraham continues and brings other opinions
- the Shevut Yaakov was asked regarding a patient who would, according to his doctors, die within a day or two, but there was one more treatment which could cure him. However if the treatment failed he would die within two hours. May one take the treatment and endanger the few days left in his life? The Shevut Yaakov answered that if there was a possibility that he would be completely cured by this treatment, one certainly must not consider the few days left. However, before taking a decision, the doctor should consider the situation carefully and consult with other doctors in the city; the decision must be taken by a two-thirds majority and the Rav of the city must approve of the final decision
- the Achiezer was asked about a patient who has six months to live according to specialists. If operated on, it was possible he would live a normal life span, but the operation was very risky and there was a greater chance of him dying earlier. He replied from the above gemara that one could deduce that where he would otherwise certainly die, we do not fear for his short life expectancy if there is a remote possibility that he can be cured and his doctors had despaired of any other possibility. One must however receive permission from the Beit Din and operate only after much thought and consultation with the best specialists
See the original for further comments and cases.
Since you are specifically interested in the writings of R Moshe Feinstein on the topic, I recommend Responsa of Rav Moshe Feinstein: Care of the Critically Ill by R Moshe David Tendler which translates key responsa on the topic.
See in particular Igrot Moshe CM II:73 regarding short-term prolongation of life in cases where further improvement is not possible as well as coercing a patient to accept a treatment against his will. In the latter (which is related to your question), R Moshe writes one is required to coerce the patient (without employing physical force) to accept treatment if the medical consensus holds that it would benefit him and may lead to a cure. However if the treatment is risky (your case), such as a surgical procedure that involves some possibility of mortal danger, coercion should not be applied if the patient refuses treatment.
See also the next responsum CM II:74 where R Moshe comments on the gemara in Avoda Zara 27b and rules that, if there is >50% probability of success (according to specialist doctors), one is obligated to get treated, below that one is permitted to (see also his responsum YD III:36).
For more, see also Practical medical halacha from Fred Rosner and R Moshe D Tendler (chapter Hazardous therapy and medical experimentation) and Judaism and healing from R J. David Bleich (chapter Hazardous procedures and experimentation).