You can do this by setting the DialogBoxLauncherTool property of the RibbonGroup to an instance of a ButtonTool.
e.g.
<igRibbon:RibbonGroup Caption="Font">
<igRibbon:RibbonGroup.DialogBoxLauncherTool>
<igRibbon:ButtonTool Caption="Font Dialog" />
</igRibbon:RibbonGroup.DialogBoxLauncherTool>
</igRibbon:RibbonGroup>