I think I can answer a lot of this question myself.
Geography, of course, played a major role:
- In the south, the Sahara put a powerful brake on the religion's expansion further into Africa. (Ethiopia, a vigorous Christian state in the Horn of Africa, also helped to slow things down.)
- In the west, despite some remarkable early successes, the Islamic world clearly overextended itself by crossing the Straits of Gibraltar into Spain, and were forced back into Africa over several centuries.
On the Islamic world's northern frontier, in Eastern Europe, the geography-driven narrative starts to break down. But this is a story I know very well: despite significant instability, as well as some astonishing moments of political and military stupidity, the rickety Byzantine Empire proved to be a surprisingly tenacious opponent of Islam. And, on the demise of that state in 1453, the Habsburgs and Russians slowed down and, eventually, largely reversed the spread of Islam into Europe.
(Please tell me if you disagree vehemently with the above!)
What I don't understand, however, is why Islam didn't spread any further to the EAST. The Rashidun Caliphate conquered Sassanian Persia with an ease which shocked everyone - not least the Muslims. From there, Islam became the dominant religion, not just in Persia proper, but throughout "Greater Iran": the Caucasus, Afghanistan and Central Asia. But, to date, the Islamic world has been incapable of repeating those kind of successes in the Indian Subcontinent. Military progress was slow: Aurangzeb came close, but never quite succeeded, in subduing all of South Asia. And Muslims have always been a minority amongst the Subcontinent's inhabitants. Why?
I know that Islam was able to bypass - sort of - its failure in India, and the religion spread to Malaysia and Indonesia - mostly through trade, I'm told - where it became dominant. But Muslim merchants must have taken their wares to Thailand, China, etc, too. Why didn't it spread to those nations?
What I don't understand, however, is why Islam didn't spread any further to the EAST.
I would say this question can be answered in two words. The Mongols. 8th century to the 14th century, the Islamic world came under significant pressure from one of the most sucessful military expansionist empires the world has ever seen. The Mongols basically up-ended and splintered Islam. They conquered significant population centers in Iraq, Persia, Syria and Turkey, including the seat of Islamic Power, their capital for 500 years, Baghdad (1258). The Islamic golden age basically was snuffed out at it's height with the fall of Baghdad and the sacking of the House of Wisdom.
Mongol invasions and conquests : Middle East
The Mongols conquered, by battle or voluntary surrender, the areas of present-day Iran, Iraq, the Caucasus, and parts of Syria and Turkey, with further Mongol raids reaching southwards into Palestine as far as Gaza in 1260 and 1300. The major battles were the Siege of Baghdad (1258), when the Mongols sacked the city which had been the center of Islamic power for 500 years, and the Battle of Ain Jalut in 1260, when the Muslim Kipchak Mamluks were able to defeat the Mongols in the battle at Ain Jalut in the southern part of the Galilee-the first time the Mongols had been decisively stopped. One thousand northern Chinese engineer squads accompanied the Mongol Khan Hulagu during his conquest of the Middle East
The Otomman Caliphate which followed successfully conquered the Middle East but never successfully unified Islamic Empire which came before. The Sunni Ottoman's never successfully conquered/rejoined with the Shia Persians for example, see Ottoman Persian Wars(16th-19th centuries). That's why the Ottoman's never really spread east. The Mongols had left them splintered, and nearly continuous wars fought between Persia and the Turkish Ottomans for three centuries left them preoccupied and without the resources of the unified Caliphate which had come before.
As most other borders these are results of long, hard centuries of wars, beginning with Islamic conquests. The border on the strait of Gibraltar is the result of 780 years of Reconquista. The borders in Eastern Europe are the result of the Turkish conquest of the Eastern Roman empire, and later conquest of Hungary, with the following almost continuous struggle which ended (or almost ended) only in the 20th century. In Cyprus and Palestine, the tension continues. The equilibrium between India/Pakistan/Bangladesh was reached only during my lifetime, and it is still unstable. The struggle in Africa and China continues. And so on.