Hi Thor; Welcome to the exciting world of Rocket science!
Folks who can easily get and afford refractory cement use it for solidity.
The downside to using refractory is the permanence. It does not come apart very easy and it can not be reused.
When you use fireclay . Any sudden impact can break apart your build.
On the plus side of using fireclay. You can disassemble your RMH
anytime. Hit it with a mallet, scrape off the bricks & simply save and rehydrate the fire clay mix to reuse
EDIT) I should add that critical areas near the firebox should be smooth clay slip.
Also using fireclay rather than cement. You need very thin mortar joints , just enough
to fill any irregularity's. It is not like cement.
Cement of course
you would use a 1/4" of mortar on every joint.