Jan Rysavy

I'm sorry, it works now, probably user error on my side.

                       Free Page Map                        
============================================================
Block 1 (
  1000: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  |................................|
  1020: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  |................................|
  1040: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  |................................|
  1060: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000  |................................|
  1080: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 FEFFFFFF  |................................|
  10A0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  10C0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  10E0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  1100: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|

Unfortunately, there seems to be another problem:

                          Summary                           
============================================================
  Block Size: 4096
  Number of blocks: 1250
  Number of streams: 648
  Signature: 0
  Age: 1
  GUID: {C56191BB-2E54-49D9-AEF4-8A1A80A664CF}
  Features: 0x1
  Has Debug Info: true
  Has Types: true
  Has IDs: true
  Has Globals: true
  Has Publics: true
  Is incrementally linked: false
  Has conflicting types: false
  Is stripped: false


                          Streams                           
============================================================
  Stream   0 (      0 bytes): [Old MSF Directory]
              Blocks: []
  Stream   1 (     75 bytes): [PDB Stream]
              Blocks: [901]
  Stream   2 (     56 bytes): [TPI Stream]
              Blocks: [902]
  Stream   3 (  84775 bytes): [DBI Stream]
              Blocks: [1225, 1226, 1227, 1228, 1229, 1230, 1231, 1232, 1233, 1234, 1235, 1236, 1237, 1238, 1239, 1240, 1241, 1242, 1243, 1244, 1245]
  Stream   4 (     56 bytes): [IPI Stream]
              Blocks: [1246]
  Stream   5 (     28 bytes): [Module "System"]
              Blocks: [4]
  Stream   6 (   2392 bytes): [Module "SysInit"]
              Blocks: [5]
  Stream   7 (     28 bytes): [Module "System.Types"]
              Blocks: [6]
  Stream   8 (     28 bytes): [Module "System.UITypes"]
              Blocks: [7]
  Stream   9 (   7520 bytes): [Module "Winapi.Windows"]
              Blocks: [8, 9]
  Stream  10 (     88 bytes): [Module "Winapi.PsAPI"]
....
  Stream 643 (    212 bytes): [Module "* Linker *"]
              Blocks: [898]
  Stream 644 (   4831 bytes): [Named Stream "/names"]
              Blocks: [899, 900]
  Stream 645 (1015700 bytes): [Symbol Records]
              Blocks: [903, 904, 905, 906, 907, 908, 909, 910, 911, 912, 913, 914, 915, 916, 917, 918, 919, 920, 921, 922, 923, 924, 925, 926, 927, 928, 929, 930, 931, 932, 933, 934, 935, 936, 937, 938, 939, 940, 941, 942, 943, 944, 945, 946, 947, 948, 949, 950, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965, 966, 967, 968, 969, 970, 971, 972, 973, 974, 975, 976, 977, 978, 979, 980, 981, 982, 983, 984, 985, 986, 987, 988, 989, 990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012, 1013, 1014, 1015, 1016, 1017, 1018, 1019, 1020, 1021, 1022, 1023, 1024, 1025, 1026, 1027, 1028, 1029, 1030, 1031, 1032, 1033, 1034, 1035, 1036, 1037, 1038, 1039, 1040, 1041, 1042, 1043, 1044, 1045, 1046, 1047, 1048, 1049, 1050, 1051, 1052, 1053, 1054, 1055, 1056, 1057, 1058, 1059, 1060, 1061, 1062, 1063, 1064, 1065, 1066, 1067, 1068, 1069, 1070, 1071, 1072, 1073, 1074, 1075, 1076, 1077, 1078, 1079, 1080, 1081, 1082, 1083, 1084, 1085, 1086, 1087, 1088, 1089, 1090, 1091, 1092, 1093, 1094, 1095, 1096, 1097, 1098, 1099, 1100, 1101, 1102, 1103, 1104, 1105, 1106, 1107, 1108, 1109, 1110, 1111, 1112, 1113, 1114, 1115, 1116, 1117, 1118, 1119, 1120, 1121, 1122, 1123, 1124, 1125, 1126, 1127, 1128, 1129, 1130, 1131, 1132, 1133, 1134, 1135, 1136, 1137, 1138, 1139, 1140, 1141, 1142, 1143, 1144, 1145, 1146, 1147, 1148, 1149, 1150]
  Stream 646 (     16 bytes): [Global Symbol Hash]
              Blocks: [1151]
  Stream 647 ( 296012 bytes): [Public Symbol Hash]
              Blocks: [1152, 1153, 1154, 1155, 1156, 1157, 1158, 1159, 1160, 1161, 1162, 1163, 1164, 1165, 1166, 1167, 1168, 1169, 1170, 1171, 1172, 1173, 1174, 1175, 1176, 1177, 1178, 1179, 1180, 1181, 1182, 1183, 1184, 1185, 1186, 1187, 1188, 1189, 1190, 1191, 1192, 1193, 1194, 1195, 1196, 1197, 1198, 1199, 1200, 1201, 1202, 1203, 1204, 1205, 1206, 1207, 1208, 1209, 1210, 1211, 1212, 1213, 1214, 1215, 1216, 1217, 1218, 1219, 1220, 1221, 1222, 1223, 1224]



                       Free Page Map                        
============================================================
Block 1 (
  1000: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  1020: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  1040: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  1060: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  1080: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  10A0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  10C0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  10E0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  1100: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  1120: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
...
  1F60: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  1F80: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  1FA0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  1FC0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  1FE0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
)



                        Stream Data                         
============================================================
  Stream 0: Old MSF Directory (dumping 0 / 0 bytes)
    Data (
    )
  Stream 1: PDB Stream (dumping 75 / 75 bytes)
    Data (
      385000: 942E3101 00000000 01000000 BB9161C5 542ED949 AEF48A1A 80A664CF 07000000  |..1...........a.T..I......d.....|
      385020: 2F6E616D 65730001 00000002 00000001 00000002 00000000 00000000 00000084  |/names..........................|
      385040: 02000000 00000041 913201                                                 |.......A.2.|
    )
  Stream 2: TPI Stream (dumping 56 / 56 bytes)
    Data (
      386000: 0BCA3101 38000000 00100000 00100000 00000000 FFFFFFFF 04000000 FFFF0300  |..1.8...........................|
      386020: 00000000 00000000 00000000 00000000 00000000 00000000                    |........................|
    )

1 minute ago, Anders Melander said:

Btw, I think I now know what checkInvariants is doing.

What threw me was that it appeared to be modifying the FPM and I couldn't understand the purpose of that since it would have made the FPM unusable afterwards. I think it's working on a copy of the FPM; It marks all blocks found in a stream as free (in the FPM copy) and once it has done that for all the streams it checks if all blocks are now marked free.

Exactly, I saw it in debugger...

Well done, you're only one bit away from success!

The checkInvariants function is now definitely different from the GitHub version. For example, they call the FPM::add method to combine two FPMs. And they do other tests that didn't exist in the GitHub version.

On offset 0x1000 set bit 0x00000008:

Success:

It looks like you are using 1 for used and 0 for free block in FPM.

According to https://github.com/microsoft/microsoft-pdb/blob/805655a28bd8198004be2ac27e6e0290121a5e89/PDB/msf/msf.cpp#L367

and https://llvm.org/docs/PDB/MsfFile.html#the-free-block-map

1 should mean free block. See attached stripped_pdb.zip:

                       Free Page Map                        
============================================================
Block 1 (
  1000: E080F8FF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  1020: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|
  1040: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF  |................................|

stripped_pdb.zip

29 minutes ago, Anders Melander said:

:)))))))))))))))))

There are these two loops visible in disassembler: https://github.com/microsoft/microsoft-pdb/blob/805655a28bd8198004be2ac27e6e0290121a5e89/PDB/msf/msf.cpp#L1395

        fpmInUse.fEnsureRoom(pnMac());
        fpmInUse.clearAll();
        UNSN    snMac = st.snMac();
        for (UNSN sn = 0; sn < snMac; sn++) {
            SI si = st.mpsnsi[sn];
            if (!si.isValid())
                continue;
            for (USPN spn = 0; spn < si.spnMac(lgCbPg()); spn++) {
                UPN pn = si.mpspnpn[spn];
                assert(validPn(pn));
                assert(!fpm.isFreePn(pn));
                assert(!fpmFreed.isFreePn(pn));
                assert(!fpmInUse.isFreePn(pn));
                fpmInUse.freePn(pn);
            }
        }

In our case we fail with sn == 0 (first stream) and spn == 0 (first page of first stream) - we don't repeat any loop.

55 minutes ago, Anders Melander said:

No, I think the problem is that I'm marking a block as allocated even though it's free.

I agree, I came to the same conclusion. Function checkInvariants fails because our FPM bit-field isn't "correct".

If you can find out where in the PDB file the bit address (i.e. the block number) being tested comes from, I should be able to narrow the cause down even more.

I don't understand exactly what you mean, can you elaborate on this point? I'd be happy to help...

    native_word mppnmask(unsigned pn) {
        return native_word(1) << (pn & (BPW-1));
    }

BPW = sizeof(native_word) * CHAR_BIT, in our (x64) case BPW = 8 * 8 = 0x40

BPW - 1 = 0x3F, another constant from loc_1800FCE71:

00007ffa`70fece73 83e03f         and     eax, 3Fh

In loc_1800FCE71 we are ?maybe? looking at inlined (Visual C++ Release build optimization) function isFreePn:

https://github.com/microsoft/microsoft-pdb/blob/805655a28bd8198004be2ac27e6e0290121a5e89/PDB/msf/msf.cpp#L367

5 hours ago, Anders Melander said:

If it's blocking the bitmap into qwords then that might be the problem. I'm blocking it into bytes.

In memory they are accessing FPM bit-field using QWORDs in x64 and DWORDs in x86, see  Array<native_word> rgw - native_word is defined based on x64/x86 build

https://github.com/microsoft/microsoft-pdb/blob/805655a28bd8198004be2ac27e6e0290121a5e89/PDB/msf/msf.cpp#L300

#if defined(_M_IA64) || defined(_M_AMD64)
    // REVIEW: ensure that using 64-bit ints on win64 doesn't impact
    // reading/writing pdbs written/read from win32.
    //
    typedef unsigned __int64    native_word;
    enum {
        lgBPW   = 6,
    };
#else
    typedef unsigned long       native_word;
    enum {
        lgBPW   = 5,
    };
#endif

    enum {
        BPW     = sizeof(native_word) * CHAR_BIT,
        iwMax   = ::cbFpmMaxBigMsf/sizeof(native_word),
        BPL     = BPW,
    };

    unsigned    iwMac;      // set to max size of the FPM bit array, based on small or big msfs
    unsigned    iwRover;    // heuristic approach for finding a free page quickly
    CB          cbPg;
    bool        fBigMsf;

    Array<native_word>
                rgw;
    native_word wFill;  

So in our case (x64) lgBPW is 6 and following function

    unsigned mppniw(unsigned pn) {
        return pn >> lgBPW;
    }

is inlined in loc_1800FCE71 as instruction

00007ffa`70fece7c 48c1e806       shr     rax, 6

21 minutes ago, Attila Kovacs said:

if it's compiled in debug mode, did they ship the own pdb? 😉

I don't think msdia140.dll is compiled in debug mode. PDB is available on Microsoft Symbol servers.

Another idea: I found what part of PDB they are reading in loc_1800FCE71, instruction mov     rax, qword ptr [r14+rax*8].

It is on offset 0x1008 of attached PDB, see TestMAP2PDB.zip. Does it help?

Edit: Sorry, marked two QWORDs in memory dump.

TestMAP2PDB.zip

OK, it looks like different problem.

Do you have any idea what they are testing in checkInvariants  / loc_1800FCE71?

Latest llvm-pdbutil (16.0.4) "dump --all" returns following error on MAP2PDB PDB:

Unexpected error processing modules: PDB does not contain the requested image section header type

Is that normal?

It looks like the checkInvariants function is more complicated in the current msdia140.dll version (14.36.32532.0) compared to the GitHub source code https://github.com/microsoft/microsoft-pdb/blob/805655a28bd8198004be2ac27e6e0290121a5e89/PDB/msf/msf.cpp#L1385

I compiled MAP2PDB from your Bugfix/LargePDB branch, exported PDB and debugged through checkInvariants. Green/red boxes is executed code. See checkInvariants.zip for details.

Neither loop is repeated in the case of my simple PDB.

loc_1800FCE71 is where checkInvariants fails.

Edit: tested on MSVC PDB where checkInvariants succeeds, exactly same code is executed, first difference is in loc_1800FCE71 where MSVC PDB has al = 0x00:

MAP2PDB PDB
00007ffa`70fece71 8bc2           mov     eax, edx
eax = 0x00000053
00007ffa`70fece73 83e03f         and     eax, 3Fh
eax = 0x00000013
00007ffa`70fece76 0fb6c8         movzx   ecx, al
ecx = 0x00000013
00007ffa`70fece79 488bc2         mov     rax, rdx
rax = 0x0000000000000053
00007ffa`70fece7c 48c1e806       shr     rax, 6
rax = 0x0000000000000001
00007ffa`70fece80 498b04c6       mov     rax, qword ptr [r14+rax*8]
rax = 0x00000000001fffff
00007ffa`70fece84 480fa3c8       bt      rax, rcx
00007ffa`70fece88 0f92c0         setb    al
al = 0x01
00007ffa`70fece8b 84c0           test    al, al
00007ffa`70fece8d 0f8507020000   jne     msdia140!MSF_HB::checkInvariants+0x42a (7ffa70fed09a)


MSVC PDB
00007ffa`6769ce71 8bc2           mov     eax, edx
eax = 0x00000420
00007ffa`6769ce73 83e03f         and     eax, 3Fh
eax = 0x00000020
00007ffa`6769ce76 0fb6c8         movzx   ecx, al
ecx = 0x00000020
00007ffa`6769ce79 488bc2         mov     rax, rdx
rax = 0x0000000000000420
00007ffa`6769ce7c 48c1e806       shr     rax, 6
rax = 0x0000000000000010
00007ffa`6769ce80 498b04c6       mov     rax, qword ptr [r14+rax*8]
rax = 0xfffffff800000760
00007ffa`6769ce84 480fa3c8       bt      rax, rcx
00007ffa`6769ce88 0f92c0         setb    al
al = 0x00
00007ffa`70fece8b 84c0           test    al, al
00007ffa`6769ce8d 0f8507020000   jne     msdia140!MSF_HB::checkInvariants+0x42a (7ffa6769d09a)

checkInvariants.zip

CONFIRMED!

Patched version of amplxe_msdia140!MSF_HB::checkInvariants (version 14.34.31942.0 from VTune 2023.1) works fine with MAP2PDB PDBs.

amplxe_msdia140.zip

Yes, exactly. I'm using WinDbg Preview from https://apps.microsoft.com/store/detail/windbg-preview/9PGJGD53TN86

Set breakpoint: bp msdia140!CDiaDataSource::loadDataFromPdb

Run 'wt' trace command: wt -m msdia140 -oR

Btw, see attached dumps. Both are using msdia140.dll version 14.36.32532.0.
msdia140_14_36_32532_error.txt is loading PDB from MAP2PDB, while

msdia140_14_36_32532_success.txt is loading PDB from MSVC simple application.

Look at difference in msdia140!MSF_HB::checkInvariants return value rax = 0 vs rax = 1.

msdia140_diff.zip

15 hours ago, Anders Melander said:

StrmTbl::~StrmTbl

Lots of calls to this.

Just one note regarding 'wt' command output. There is only one call to StrmTbl::~StrmTbl (etc.), see summary in dump :

Function Name                               Invocations MinInst MaxInst AvgInst

msdia140!StrmTbl::~StrmTbl                            1     597     597     597

'wt' output explained: https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/wt--trace-and-watch-data-

LocalFree is simply called in a loop:

   28     0 [  7]               msdia140!StrmTbl::~StrmTbl
    3     0 [  8]                 msdia140!operator delete
    1     0 [  8]                 KERNEL32!LocalFreeStub
   23     0 [  8]                 KERNELBASE!LocalFree
   40    27 [  7]               msdia140!StrmTbl::~StrmTbl
    3     0 [  8]                 msdia140!operator delete
    1     0 [  8]                 KERNEL32!LocalFreeStub
   23     0 [  8]                 KERNELBASE!LocalFree
   52    54 [  7]               msdia140!StrmTbl::~StrmTbl
    3     0 [  8]                 msdia140!operator delete
    1     0 [  8]                 KERNEL32!LocalFreeStub
   23     0 [  8]                 KERNELBASE!LocalFree

15 minutes ago, Anders Melander said:

PortablePDB::PortablePDB

Something wrong here. "Portable PDB" is the .NET PDB format. It's a completely different file format.
I'm guessing it's falling back to that format after failing to validate the file as PDB.

Yes, see attached 'wt' output from old msdia140.dll 14.29.30035.0 for the same input PDB file. In this case loadDataFromPdb succeeds.

msdia14_142930035.zip

4 hours ago, Anders Melander said:

Does it produce any debug output while loading?

Seems there is some logger framework, probably undocumented. In msdia140.zip is attached 'wt' command output for msdia140!CDiaDataSource::loadDataFromPdb.

msdia140.zip

MS released debug symbols for msdia140.dll... nice

26 minutes ago, shineworld said:

Can you attach the working DLL to try ?

 

msdia140.zip

I tried to load the PDB using the current version of C:\Program Files\Microsoft Visual Studio\2022\Community\DIA SDK (14.36.32532.0) and the method loadDataFromPdb returns the error E_PDB_FORMAT.

The same test with older msdia140.dll (14.29.30035.0) works.

VTune 2023.1.0 works fine also with C:\Program Files\AMD\AMDuProf\bin\msdia140.dll (version 14.29.30035.0).

This also probably explains why Profiler in Visual Studio (2022) stopped displaying function names in recent versions. It uses newer versions of msdia140.dll.