The FileExists() check is unnecessary and should be removed. It introduces a race condition (the file may exist before FileExists() is called, and then gets deleted before CreateFile() is called) as well as a possible failure point (FileExists() is not 100% accurate in all possible situations, there are cases where it returns a wrong result). Calling CreateFile() by itself with OPEN_EXISTING is good enough. If the file really does not exist, CreateFile() will fail with an ERROR_FILE_NOT_FOUND error code.
Also, your code does not take ERROR_SHARING_VIOLATION into account, which would mean the file is actually in use but you don't have access to open it.
Try this instead:
function IsFileInUse(FileName: TFileName): Boolean;
var
HFileRes: THandle;
begin
HFileRes := CreateFile(PChar(FileName),
GENERIC_READ or GENERIC_WRITE,
0,
nil,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
0);
if (HFileRes <> INVALID_HANDLE_VALUE) then
begin
CloseHandle(HFileRes);
Result := False;
end else
Result := (GetLastError() = ERROR_SHARING_VIOLATION);
end;