I suspect the main reason this occurs as "smelly" is because there are so few reasons to do this kind of thing in Delphi. It's far more common in c/c++ as old-timers think it's perfectly acceptable.
if you have a justifiable need to do bit-whacking like this, then I say, do it. I probably wouldn't do it for, say, a mere 5% performance gain, but 50%+ is far more justifiable.
It would be nice to see some comments as to what's supposed to be going on because average Delphi coders won't make heads or tails of the code. (I personally have trouble dealing with pointers in Delphi, since I do it so seldom.)
And be especially careful to point out any dependencies on assumed sizes of vars, especially integers that can vary depending on the platform.
(Better yet, use explicit types that ensure fixed sizes and explain why.)