Anders Melander

In my opinion, the online help is better. I bought the book a few years ago when I was about to transition a system from ADO to FireDAC and I found the book to be completely useless. It covers the basics, but so does the help and the help is up to date. Advanced topics are barely covered at all.

I would suggest you add it as a Git submodule.

Yes, it looks like you used the old version (I hope 🙂).

Just did: https://bitbucket.org/anders_melander/map2pdb/downloads/ (map2pdb.20230526a-3.0.1.zip) Updating the readme.md now...

Success! Pushed and merged to master. Thank you to all that helped. You really saved me a lot of time. Especially since I was pretty unfocused and preoccupied (new job, lots of other projects, summer finally arrived).

Ah... That must be this block: // Write an empty block just so our layout matches LLVM. // This can be removed once our output is validated. FWriter.BeginBlock; FWriter.Write(Byte(0)); FWriter.EndBlock; The block doesn't appear in any stream so it must be marked free. It's block #3 = bit 3 = 0x00000008 I'll try to remove the block instead first. Stay tuned... Btw, I think I now know what checkInvariants is doing. What threw me was that it appeared to be modifying the FPM and I couldn't understand the purpose of that since it would have made the FPM unusable afterwards. I think it's working on a copy of the FPM; It marks all blocks found in a stream as free (in the FPM copy) and once it has done that for all the streams it checks if all blocks are now marked free. And well done, yourself! This looks interesting 🙂

Unfortunately, flipping the meaning of the bit didn't solve the problem. Same error message but it's probably another cause this time. I have also tried using cvdump and that fails too; Unsurprising since it's also using msdia140.dll.

It's an easy fix. I'll do it tomorrow. Thanks.

Output from : llvm-pdbutil dump -summary -streams -stream-blocks <pdbfile> Summary ============================================================ Block Size: 4096 Number of blocks: 85 Number of streams: 47 Signature: 0 Age: 1 GUID: {CDAF4F74-CCFB-6F40-A551-C2BEE853FBEF} Features: 0x1 Has Debug Info: true Has Types: true Has IDs: true Has Globals: true Has Publics: true Is incrementally linked: false Has conflicting types: false Is stripped: false Streams ============================================================ Stream 0 ( 0 bytes): [Old MSF Directory] Blocks: [] Stream 1 ( 75 bytes): [PDB Stream] Blocks: [65] Stream 2 ( 56 bytes): [TPI Stream] Blocks: [66] Stream 3 ( 5527 bytes): [DBI Stream] Blocks: [80, 81] Stream 4 ( 56 bytes): [IPI Stream] Blocks: [82] Stream 5 (58764 bytes): [Module "System"] Blocks: [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18] Stream 6 ( 2392 bytes): [Module "SysInit"] Blocks: [19] Stream 7 ( 56 bytes): [Module "System.Types"] Blocks: [20] Stream 8 ( 64 bytes): [Module "System.UITypes"] Blocks: [21] Stream 9 ( 1132 bytes): [Module "Winapi.Windows"] Blocks: [22] Stream 10 ( 56 bytes): [Module "System.SysConst"] Blocks: [23] Stream 11 ( 88 bytes): [Module "Winapi.PsAPI"] Blocks: [24] Stream 12 ( 56 bytes): [Module "System.RTLConsts"] Blocks: [25] Stream 13 (12756 bytes): [Module "System.Character"] Blocks: [26, 27, 28, 29] Stream 14 ( 28 bytes): [Module "System.Internal.ExcUtils"] Blocks: [30] Stream 15 (24016 bytes): [Module "System.SysUtils"] Blocks: [31, 32, 33, 34, 35, 36] Stream 16 ( 132 bytes): [Module "TestMAP2PDB"] Blocks: [37] Stream 17 ( 248 bytes): [Module "System"] Blocks: [38] Stream 18 ( 72 bytes): [Module "SysInit"] Blocks: [39] Stream 19 ( 72 bytes): [Module "System.UITypes"] Blocks: [40] Stream 20 ( 88 bytes): [Module "Winapi.Windows"] Blocks: [41] Stream 21 ( 72 bytes): [Module "Winapi.PsAPI"] Blocks: [42] Stream 22 ( 72 bytes): [Module "System.Character"] Blocks: [43] Stream 23 ( 64 bytes): [Module "System.Internal.ExcUtils"] Blocks: [44] Stream 24 ( 864 bytes): [Module "System.SysUtils"] Blocks: [45] Stream 25 ( 136 bytes): [Module "TestMAP2PDB"] Blocks: [46] Stream 26 ( 28 bytes): [Module "System"] Blocks: [47] Stream 27 ( 28 bytes): [Module "SysInit"] Blocks: [48] Stream 28 ( 28 bytes): [Module "Winapi.Windows"] Blocks: [49] Stream 29 ( 28 bytes): [Module "System.Internal.ExcUtils"] Blocks: [50] Stream 30 ( 28 bytes): [Module "System.SysUtils"] Blocks: [51] Stream 31 ( 28 bytes): [Module "System"] Blocks: [52] Stream 32 ( 28 bytes): [Module "SysInit"] Blocks: [53] Stream 33 ( 28 bytes): [Module "System.UITypes"] Blocks: [54] Stream 34 ( 28 bytes): [Module "Winapi.Windows"] Blocks: [55] Stream 35 ( 28 bytes): [Module "Winapi.PsAPI"] Blocks: [56] Stream 36 ( 28 bytes): [Module "System.Character"] Blocks: [57] Stream 37 ( 28 bytes): [Module "System.Internal.ExcUtils"] Blocks: [58] Stream 38 ( 28 bytes): [Module "System.SysUtils"] Blocks: [59] Stream 39 ( 28 bytes): [Module "TestMAP2PDB"] Blocks: [60] Stream 40 ( 28 bytes): [Module "System"] Blocks: [61] Stream 41 ( 28 bytes): [Module "System.SysUtils"] Blocks: [62] Stream 42 ( 212 bytes): [Module "* Linker *"] Blocks: [63] Stream 43 ( 436 bytes): [Named Stream "/names"] Blocks: [64] Stream 44 (30880 bytes): [Symbol Records] Blocks: [67, 68, 69, 70, 71, 72, 73, 74] Stream 45 ( 16 bytes): [Global Symbol Hash] Blocks: [75] Stream 46 (15288 bytes): [Public Symbol Hash] Blocks: [76, 77, 78, 79]

By the way, stream #0 is reserved for "a copy of the previous version of the stream directory". It should be empty, so there is no "first page" in it.

I can't match that with my assumption that I've incorrectly marked a block as allocated (i.e. the corresponding bit is set). As far as I can tell that code verifies that the block number is valid and that the block is marked as allocated. I have no idea about what the call to FreePn does.

Like you did here: I'm assuming the function that tests the bit is passed a block number which it then translates to an offset into a qword array (BlockNum and (not 63)) and a bit number into the qword (BlockNum and 63). Where does this block number come from? I.e. if you could apply your magic 🧙 and find out what offset in the PDB file is supplying the block number then we will have a better idea of what the problem is.

I'm surprised that something like ERP would benefit much from that. Do you have any numbers? Apart from that, like David, I have good experience with madExcept: No problems with optimization being on.

Yes. I understood the scheme; I was more concerned about if we are addressing the same bits. What got me a little confused is that I'm doing a lot of work with big-endian data (TrueType fonts) on another project at the moment where the QWORD/BYTE size would have mattered. However, since we are both using little-endian addressing that's not a problem here. No, I think the problem is that I'm marking a block as allocated even though it's free. When setting the bits that mark a block as allocated I'm taking advantage of the fact that I have no free blocks in the physical file. I write everything sequentially and only once. Since I keep a count of how many blocks the file contains (that's recorded in the header/superblock at the end), I can simply set the required amount of bits once everything has been written. One possible problem here is that I might be using the wrong block concept for the block numbers in the bitmap. There are physical blocks that have a direct and linear mapping to their position in the file and then there are logical blocks that exclude stuff like the superblock and the free page maps (FPM). Since the position of the superblock and the FPMs are fixed it would make sense to exclude them from the bitmap (i.e. it maps to logical blocks). I can try and see if it makes any difference to use logical instead of physical block numbers in the FPM. In your test case, the bit in the MAP2PDB PDB is set (i.e. block is allocated) and in the MSVC PDB the bit is clear (i.e. block not allocated). We cannot expect the same bits to be set since we are storing different data in the two files, however, since it seems the failure occurs because the bit is set in our file msdia140.dll apparently expect the bit to be clear so the problem must be that I have marked a block as allocated even though it doesn't appear in any stream (or rather in any stream directory). If you can find out where in the PDB file the bit address (i.e. the block number) being tested comes from, I should be able to narrow the cause down even more.

I can't see how. Is there anything, in particular, you're hinting at? I've read that document many, many times. It's a good starting point but it's very incomplete and, in some cases, plain wrong.

As far as I can tell it's the Free Page Map. This matches with the fact that the asm is doing bit tests. If it's blocking the bitmap into qwords then that might be the problem. I'm blocking it into bytes. To mark block 1 as allocated I would set bit 0 of the first byte, block 9 would be bit 0 of the second byte, etc. It could be that I should be using another bit-layout. It could also be that I'm simply not marking all the blocks allocated that I should - or too many.

Yes, it definitely should. I'm just too tired to dissect the file in a hex editor right now 🙂 Maybe later tonight.

No. I can't figure it out.

llvm-pdbutil.exe version 9.0.0.0 doesn't fail with the new PDBs. However, with the large blocksize (map2pdb -blocksize:8192 ...) it fails as expected (it's documented to only support a blocksize of 4096): The source of that error message is here: https://github.com/llvm/llvm-project/blob/f74bb326949aec0cac8e54ff00fc081f746ff35d/llvm/tools/llvm-pdbutil/DumpOutputStyle.cpp#L395 It would have been nice if the message specified what header type it was looking for but In any case, it's a bit suspicious that it mentions "image section header" because that's a thing in the PE image and the PDB doesn't contain any PE related stuff. It could just be a bad choice of words. Anyway, moving along. So the error message is produced by loadSectionHeaders(type). loadSectionHeaders(type) is called from dumpSectionHeaders(type) which is called from dumpSectionHeaders() with type=SectionHdr and type=SectionHdrOrig. I have neither type implemented but I have a comment in my source about SectionHdr: function TDebugInfoPdbWriter.EmitDBISubstreamDebugHeader(Writer: TBinaryBlockWriter): Cardinal; begin Result := Writer.Position; // We don't provide any of the optional debug streams yet for var HeaderType := Low(PDBDbgHeaderType) to High(PDBDbgHeaderType) do Writer.Write(Word(TMSFStream.NullStreamIndex)); // TODO : I believe the SectionHdr stream contains the segment names (* for var Segment in FDebugInfo.Segments do begin xxx.Add(Segment.SegClassName); xxx.Add(Segment.Name); end; *) Result := Writer.Position - Result; end; Since these streams are optional it's pretty stupid of llvm-pdbutil to bug out if they aren't present. The old llvm-pdbutil didn't read them (which is probably why I didn't know the format or content of the streams) but since the new one does it should be possible to deduce the format from their source. I'm just concerned that this might be a completely wasted effort if msdia140.dll is failing on something else.

No, not as far as I remember. I don't have the map2pdb project on the system I'm on right now, but when I get home I will verify that the older version of llvm-pdbutil (the version I targeted with the YAML output) doesn't fail like that.

Of course I couldn't help myself. The Bugfix/LargePDB branch contains the changes so the Free Page Map now contains the correct values. Unfortunately, that didn't solve the problem.

Wow. Thanks!

So I'm looking at checkInvariants in msf.cpp and there's this comment: MSF is the container format for a PDB file. The MSF format is pretty much a mini FAT file system, the files being the different PDB tables: Source file names, line numbers, symbols, etc. Internally the MSF file system is divided into intervals. Each interval contains <blocksize> blocks and each block is <blocksize> bytes long. The blocksize used to be 4096. Now it can apparently also be 8192. At the start of each interval, there are two blocks (the Free Page Map) that contain a bitmap of allocated blocks in the interval. Allocated as in "in-use" by a stream. A stream is just a collection of blocks. All streams are listed in the stream directory. The stream directory is stored in one or more blocks. At the start of the file is the superblock. It contains various info about the file: blocksize, index of the first Free Page Map, the number of blocks in the file, a pointer to the stream directory, etc. MSF Block and Interval layout Interval | 0 | 1 ------------+------------+------+------+------+------+------+------+------+------+------+------+------+- - - Block index | 0 | 1 | 2 | 3 | 4 | ... | 4095 | 4096 | 4097 | 4095 | 4096 | 4097 | ... ------------+------------+------+------+------+------+------+------+------+------+------+------+------+- - - Log block | 0 (N/A) | N/A | N/A | 1 | 2 | ... | ... | 4094 | N/A | N/A | 4095 | ... | ... ------------+------------+------+------+------+------+------+------+------+------+------+------+------+- - - Phys offset | 0 | 4096 | 8192 |12288 |16384 | ... | ... |4096^2|+4096 |+8192 | ... | ... | ... ------------+------------+------+------+------+------+------+------+------+------+------+------+------+- - - Content | Superblock | FPM1 | FPM2 | Data | Data | Data | Data | Data | FPM1 | FPM2 | Data | Data | Data So in theory some of the blocks in an interval can be in use and some of them can be free and those that are in use should be referenced in the higher level stream's index of MSF blocks - otherwise, they are "leaked". I believe this is what checkInvariants verifies. Now, since I'm writing the PDB file in one go and never have a need to go back and free or modify an already allocated MSF block, I always mark all blocks as allocated when I start a new interval in the file. This means that I can, and most likely will, end up with blocks marked as allocated in the bitmap but not actually in use (or physically present in the file). procedure TBinaryBlockWriter.WriteBlockMap; const NoVacancies: Byte = $FF; begin BeginBlock; Assert((BlockIndex mod FBlockSize) in [1, 2]); // Mark all BlockSize*8 blocks occupied for var i := 0 to FBlockSize-1 do FStream.WriteBuffer(NoVacancies, 1); FStreamSize := Max(FStreamSize, FStream.Position); EndBlock(True); end; So why wasn't this a problem with the old version? In the old Microsoft source checkInvariants is only active in debug builds so my guess is that the old version simply doesn't perform this validation. Anyway, it's the best guess I have right now so it should be pursued. I'm not sure when I will get time to do so though.

I want a sex change operation so I can have your children. (it means "thank you" in case you wondered)

It can.