Jan Rysavy

Also: https://wiert.me/2020/10/07/reverse-engineering-delphi-and-turbo-pascal-unit-interfaces/

This works fine, but we also need debug symbols for vcl290.bpl and vclimg290.bpl, whose projects are missing in the distribution.

Hi, Do we have any information about x64 RTL debug symbols under Delphi for Windows? For x86, MAP2PDB now supports reading from C:\Program Files (x86)\Embarcadero\Studio\23.0\bin\rtl290.jdbg, which is fantastic. However, for x64, this file doesn't seem to be present in the Delphi 11 or Delphi 12 distributions. I checked the Modules window in the debugger, and it appears that the debug symbols are being read from C:\Program Files (x86)\Embarcadero\Studio\23.0\lib\win64\debug\rtl.dcp. Do we have any details on the DCP structure, and is there any possibility of generating a .MAP file from it? Thanks!

I'm sorry, it works now, probably user error on my side. Free Page Map ============================================================ Block 1 ( 1000: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 |................................| 1020: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 |................................| 1040: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 |................................| 1060: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 |................................| 1080: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 FEFFFFFF |................................| 10A0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 10C0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 10E0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 1100: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................|

Unfortunately, there seems to be another problem: Summary ============================================================ Block Size: 4096 Number of blocks: 1250 Number of streams: 648 Signature: 0 Age: 1 GUID: {C56191BB-2E54-49D9-AEF4-8A1A80A664CF} Features: 0x1 Has Debug Info: true Has Types: true Has IDs: true Has Globals: true Has Publics: true Is incrementally linked: false Has conflicting types: false Is stripped: false Streams ============================================================ Stream 0 ( 0 bytes): [Old MSF Directory] Blocks: [] Stream 1 ( 75 bytes): [PDB Stream] Blocks: [901] Stream 2 ( 56 bytes): [TPI Stream] Blocks: [902] Stream 3 ( 84775 bytes): [DBI Stream] Blocks: [1225, 1226, 1227, 1228, 1229, 1230, 1231, 1232, 1233, 1234, 1235, 1236, 1237, 1238, 1239, 1240, 1241, 1242, 1243, 1244, 1245] Stream 4 ( 56 bytes): [IPI Stream] Blocks: [1246] Stream 5 ( 28 bytes): [Module "System"] Blocks: [4] Stream 6 ( 2392 bytes): [Module "SysInit"] Blocks: [5] Stream 7 ( 28 bytes): [Module "System.Types"] Blocks: [6] Stream 8 ( 28 bytes): [Module "System.UITypes"] Blocks: [7] Stream 9 ( 7520 bytes): [Module "Winapi.Windows"] Blocks: [8, 9] Stream 10 ( 88 bytes): [Module "Winapi.PsAPI"] .... Stream 643 ( 212 bytes): [Module "* Linker *"] Blocks: [898] Stream 644 ( 4831 bytes): [Named Stream "/names"] Blocks: [899, 900] Stream 645 (1015700 bytes): [Symbol Records] Blocks: [903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012, 1013, 1014, 1015, 1016, 1017, 1018, 1019, 1020, 1021, 1022, 1023, 1024, 1025, 1026, 1027, 1028, 1029, 1030, 1031, 1032, 1033, 1034, 1035, 1036, 1037, 1038, 1039, 1040, 1041, 1042, 1043, 1044, 1045, 1046, 1047, 1048, 1049, 1050, 1051, 1052, 1053, 1054, 1055, 1056, 1057, 1058, 1059, 1060, 1061, 1062, 1063, 1064, 1065, 1066, 1067, 1068, 1069, 1070, 1071, 1072, 1073, 1074, 1075, 1076, 1077, 1078, 1079, 1080, 1081, 1082, 1083, 1084, 1085, 1086, 1087, 1088, 1089, 1090, 1091, 1092, 1093, 1094, 1095, 1096, 1097, 1098, 1099, 1100, 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 1110, 1111, 1112, 1113, 1114, 1115, 1116, 1117, 1118, 1119, 1120, 1121, 1122, 1123, 1124, 1125, 1126, 1127, 1128, 1129, 1130, 1131, 1132, 1133, 1134, 1135, 1136, 1137, 1138, 1139, 1140, 1141, 1142, 1143, 1144, 1145, 1146, 1147, 1148, 1149, 1150] Stream 646 ( 16 bytes): [Global Symbol Hash] Blocks: [1151] Stream 647 ( 296012 bytes): [Public Symbol Hash] Blocks: [1152, 1153, 1154, 1155, 1156, 1157, 1158, 1159, 1160, 1161, 1162, 1163, 1164, 1165, 1166, 1167, 1168, 1169, 1170, 1171, 1172, 1173, 1174, 1175, 1176, 1177, 1178, 1179, 1180, 1181, 1182, 1183, 1184, 1185, 1186, 1187, 1188, 1189, 1190, 1191, 1192, 1193, 1194, 1195, 1196, 1197, 1198, 1199, 1200, 1201, 1202, 1203, 1204, 1205, 1206, 1207, 1208, 1209, 1210, 1211, 1212, 1213, 1214, 1215, 1216, 1217, 1218, 1219, 1220, 1221, 1222, 1223, 1224] Free Page Map ============================================================ Block 1 ( 1000: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 1020: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 1040: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 1060: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 1080: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 10A0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 10C0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 10E0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 1100: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 1120: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| ... 1F60: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 1F80: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 1FA0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 1FC0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 1FE0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| ) Stream Data ============================================================ Stream 0: Old MSF Directory (dumping 0 / 0 bytes) Data ( ) Stream 1: PDB Stream (dumping 75 / 75 bytes) Data ( 385000: 942E3101 00000000 01000000 BB9161C5 542ED949 AEF48A1A 80A664CF 07000000 |..1...........a.T..I......d.....| 385020: 2F6E616D 65730001 00000002 00000001 00000002 00000000 00000000 00000084 |/names..........................| 385040: 02000000 00000041 913201 |.......A.2.| ) Stream 2: TPI Stream (dumping 56 / 56 bytes) Data ( 386000: 0BCA3101 38000000 00100000 00100000 00000000 FFFFFFFF 04000000 FFFF0300 |..1.8...........................| 386020: 00000000 00000000 00000000 00000000 00000000 00000000 |........................| )

Exactly, I saw it in debugger...

Well done, you're only one bit away from success! The checkInvariants function is now definitely different from the GitHub version. For example, they call the FPM::add method to combine two FPMs. And they do other tests that didn't exist in the GitHub version. On offset 0x1000 set bit 0x00000008: Success:

It looks like you are using 1 for used and 0 for free block in FPM. According to https://github.com/microsoft/microsoft-pdb/blob/805655a28bd8198004be2ac27e6e0290121a5e89/PDB/msf/msf.cpp#L367 and https://llvm.org/docs/PDB/MsfFile.html#the-free-block-map 1 should mean free block. See attached stripped_pdb.zip: Free Page Map ============================================================ Block 1 ( 1000: E080F8FF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 1020: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| 1040: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF |................................| stripped_pdb.zip

:)))))))))))))))))

There are these two loops visible in disassembler: https://github.com/microsoft/microsoft-pdb/blob/805655a28bd8198004be2ac27e6e0290121a5e89/PDB/msf/msf.cpp#L1395 fpmInUse.fEnsureRoom(pnMac()); fpmInUse.clearAll(); UNSN snMac = st.snMac(); for (UNSN sn = 0; sn < snMac; sn++) { SI si = st.mpsnsi[sn]; if (!si.isValid()) continue; for (USPN spn = 0; spn < si.spnMac(lgCbPg()); spn++) { UPN pn = si.mpspnpn[spn]; assert(validPn(pn)); assert(!fpm.isFreePn(pn)); assert(!fpmFreed.isFreePn(pn)); assert(!fpmInUse.isFreePn(pn)); fpmInUse.freePn(pn); } } In our case we fail with sn == 0 (first stream) and spn == 0 (first page of first stream) - we don't repeat any loop.

I agree, I came to the same conclusion. Function checkInvariants fails because our FPM bit-field isn't "correct". I don't understand exactly what you mean, can you elaborate on this point? I'd be happy to help...

native_word mppnmask(unsigned pn) { return native_word(1) << (pn & (BPW-1)); } BPW = sizeof(native_word) * CHAR_BIT, in our (x64) case BPW = 8 * 8 = 0x40 BPW - 1 = 0x3F, another constant from loc_1800FCE71: 00007ffa`70fece73 83e03f and eax, 3Fh In loc_1800FCE71 we are ?maybe? looking at inlined (Visual C++ Release build optimization) function isFreePn: https://github.com/microsoft/microsoft-pdb/blob/805655a28bd8198004be2ac27e6e0290121a5e89/PDB/msf/msf.cpp#L367

In memory they are accessing FPM bit-field using QWORDs in x64 and DWORDs in x86, see Array<native_word> rgw - native_word is defined based on x64/x86 build https://github.com/microsoft/microsoft-pdb/blob/805655a28bd8198004be2ac27e6e0290121a5e89/PDB/msf/msf.cpp#L300 #if defined(_M_IA64) || defined(_M_AMD64) // REVIEW: ensure that using 64-bit ints on win64 doesn't impact // reading/writing pdbs written/read from win32. // typedef unsigned __int64 native_word; enum { lgBPW = 6, }; #else typedef unsigned long native_word; enum { lgBPW = 5, }; #endif enum { BPW = sizeof(native_word) * CHAR_BIT, iwMax = ::cbFpmMaxBigMsf/sizeof(native_word), BPL = BPW, }; unsigned iwMac; // set to max size of the FPM bit array, based on small or big msfs unsigned iwRover; // heuristic approach for finding a free page quickly CB cbPg; bool fBigMsf; Array<native_word> rgw; native_word wFill; So in our case (x64) lgBPW is 6 and following function unsigned mppniw(unsigned pn) { return pn >> lgBPW; } is inlined in loc_1800FCE71 as instruction 00007ffa`70fece7c 48c1e806 shr rax, 6

I don't think msdia140.dll is compiled in debug mode. PDB is available on Microsoft Symbol servers.

Another idea: I found what part of PDB they are reading in loc_1800FCE71, instruction mov rax, qword ptr [r14+rax*8]. It is on offset 0x1008 of attached PDB, see TestMAP2PDB.zip. Does it help? Edit: Sorry, marked two QWORDs in memory dump. TestMAP2PDB.zip

OK, it looks like different problem. Do you have any idea what they are testing in checkInvariants / loc_1800FCE71?

Latest llvm-pdbutil (16.0.4) "dump --all" returns following error on MAP2PDB PDB: Unexpected error processing modules: PDB does not contain the requested image section header type Is that normal?

It looks like the checkInvariants function is more complicated in the current msdia140.dll version (14.36.32532.0) compared to the GitHub source code https://github.com/microsoft/microsoft-pdb/blob/805655a28bd8198004be2ac27e6e0290121a5e89/PDB/msf/msf.cpp#L1385 I compiled MAP2PDB from your Bugfix/LargePDB branch, exported PDB and debugged through checkInvariants. Green/red boxes is executed code. See checkInvariants.zip for details. Neither loop is repeated in the case of my simple PDB. loc_1800FCE71 is where checkInvariants fails. Edit: tested on MSVC PDB where checkInvariants succeeds, exactly same code is executed, first difference is in loc_1800FCE71 where MSVC PDB has al = 0x00: MAP2PDB PDB 00007ffa`70fece71 8bc2 mov eax, edx eax = 0x00000053 00007ffa`70fece73 83e03f and eax, 3Fh eax = 0x00000013 00007ffa`70fece76 0fb6c8 movzx ecx, al ecx = 0x00000013 00007ffa`70fece79 488bc2 mov rax, rdx rax = 0x0000000000000053 00007ffa`70fece7c 48c1e806 shr rax, 6 rax = 0x0000000000000001 00007ffa`70fece80 498b04c6 mov rax, qword ptr [r14+rax*8] rax = 0x00000000001fffff 00007ffa`70fece84 480fa3c8 bt rax, rcx 00007ffa`70fece88 0f92c0 setb al al = 0x01 00007ffa`70fece8b 84c0 test al, al 00007ffa`70fece8d 0f8507020000 jne msdia140!MSF_HB::checkInvariants+0x42a (7ffa70fed09a) MSVC PDB 00007ffa`6769ce71 8bc2 mov eax, edx eax = 0x00000420 00007ffa`6769ce73 83e03f and eax, 3Fh eax = 0x00000020 00007ffa`6769ce76 0fb6c8 movzx ecx, al ecx = 0x00000020 00007ffa`6769ce79 488bc2 mov rax, rdx rax = 0x0000000000000420 00007ffa`6769ce7c 48c1e806 shr rax, 6 rax = 0x0000000000000010 00007ffa`6769ce80 498b04c6 mov rax, qword ptr [r14+rax*8] rax = 0xfffffff800000760 00007ffa`6769ce84 480fa3c8 bt rax, rcx 00007ffa`6769ce88 0f92c0 setb al al = 0x00 00007ffa`70fece8b 84c0 test al, al 00007ffa`6769ce8d 0f8507020000 jne msdia140!MSF_HB::checkInvariants+0x42a (7ffa6769d09a) checkInvariants.zip

CONFIRMED! Patched version of amplxe_msdia140!MSF_HB::checkInvariants (version 14.34.31942.0 from VTune 2023.1) works fine with MAP2PDB PDBs. amplxe_msdia140.zip

Yes, exactly. I'm using WinDbg Preview from https://apps.microsoft.com/store/detail/windbg-preview/9PGJGD53TN86 Set breakpoint: bp msdia140!CDiaDataSource::loadDataFromPdb Run 'wt' trace command: wt -m msdia140 -oR Btw, see attached dumps. Both are using msdia140.dll version 14.36.32532.0. msdia140_14_36_32532_error.txt is loading PDB from MAP2PDB, while msdia140_14_36_32532_success.txt is loading PDB from MSVC simple application. Look at difference in msdia140!MSF_HB::checkInvariants return value rax = 0 vs rax = 1. msdia140_diff.zip

Just one note regarding 'wt' command output. There is only one call to StrmTbl::~StrmTbl (etc.), see summary in dump : Function Name Invocations MinInst MaxInst AvgInst msdia140!StrmTbl::~StrmTbl 1 597 597 597 'wt' output explained: https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/wt--trace-and-watch-data- LocalFree is simply called in a loop: 28 0 [ 7] msdia140!StrmTbl::~StrmTbl 3 0 [ 8] msdia140!operator delete 1 0 [ 8] KERNEL32!LocalFreeStub 23 0 [ 8] KERNELBASE!LocalFree 40 27 [ 7] msdia140!StrmTbl::~StrmTbl 3 0 [ 8] msdia140!operator delete 1 0 [ 8] KERNEL32!LocalFreeStub 23 0 [ 8] KERNELBASE!LocalFree 52 54 [ 7] msdia140!StrmTbl::~StrmTbl 3 0 [ 8] msdia140!operator delete 1 0 [ 8] KERNEL32!LocalFreeStub 23 0 [ 8] KERNELBASE!LocalFree

Yes, see attached 'wt' output from old msdia140.dll 14.29.30035.0 for the same input PDB file. In this case loadDataFromPdb succeeds. msdia14_142930035.zip

Seems there is some logger framework, probably undocumented. In msdia140.zip is attached 'wt' command output for msdia140!CDiaDataSource::loadDataFromPdb. msdia140.zip

MS released debug symbols for msdia140.dll... nice

msdia140.zip