For long-term stability, you need to mix the perlite with a small amount of clay, just enough to hold the perlite granules together. With use, the clay near the inner face will fire into pottery, making it stronger as it ages.
There are two basic methods I know of: making a thin soupy clay slurry and mixing that into the perlite, or (my preference) misting the perlite (in a mixing tray) with water, dusting on some dry powdered clay, mixing, and repeating until all the perlite particles have a thin coat of clay on them. Then put the perlite-clay into your forms an inch or two at a time and firmly compact it so the particles all contact each other. A small piece of 2x4 or 2x2 is good for this. Don't ram so hard you crush the perlite, but be firm.
For this purpose, bagged fireclay from a masonry supplier would be best, as it is already perfectly fine and uniform. If you have good quality natural clay locally and purchased clay is hard to find, you can use that with careful screening to get all particles out. Some natural clays may have a low melting temperature and experience distortion on the inner face in a really good hot RMH core. Testing a bit of it in a kiln at known temperatures would be a good idea. My local clay fires nicely to cone 06 (1850F, ordinary terra cotta, earthenware temperatures) vitrifies and distorts but is very strong at cone 6 (2250F, ordinary electric kiln stoneware firing temperature) and melts at cone 10 (2350F, high-fired stoneware temperature). Most parts of an RMH will not get much above cone 6 but there could be exceptions, especially in a larger system. Fireclay will be able to stand any temperature you can reasonably throw at it.