That is not a good design. You should move the long calculation to a separate worker thread instead, and then exit the OnClick handler. Have the thread post updates to the UI thread if needed. And you can disable your UI while the thread is running, and then re-enable it when the thread is finished. Never block the UI thread, and avoid using Application->ProcessMessages() whenever possible, it causes more problems than it solves. Let the UI thread run normally.
By doing that switching, the modal Form likely lost input focus and wasn't the "active" Form anymore at the time when MessageBox() was called, which likely ended up with the MessageBox dialog becoming owned by either another Form's window, or the hidden TApplication window. That would account for why the MessageBox dialog was allowed to go behind your modal Form. The modal Form's window needs to be the owner of the MessageBox dialog in order for the dialog to stay on top of that Form. Which means, you need to either call the Win32 MessageBox() (or TaskDialog/Indirect()) function directly so you can explicitly pass in the desired owner window, or else use the TApplication/Events::OnGetActiveFormHandle event to provide the desired owner window even if the Form is not "active".