10

Trying to post answer to this question: WinDbg x64: Cannot debug a crash dump - failed to load data access DLL

Get this message:

Your post appears to contain code that is not properly formatted as code. Please indent all code by 4 spaces using the code toolbar button or the CTRL+K keyboard shortcut. For more editing help, click the [?] toolbar icon.

My post does not contain any code, it does contain debugger output that I want pre spacing, and I need to highlight several key elements out of it.

How can I turn off this limitation? I'm seriously frustrated and highly annoyed that I go to all this trouble to answer and post here, but SO won't let me. None of the other related questions appeared to help.

Here is the body of the post that is being rejected:


<!-- language: lang-none -->

I just spent the day debugging a bunch of cases where we ran into this scenario. The SOS+CLR on the same box as the crash were unable to load within WinDbg, and "lm v" reported two different versions for the same module:

<pre>0:011> lm vM *clr.dll
start             end                 module name
000007fe`f2f50000 000007fe`f38b0000   clr      # (pdb symbols)          c:\symbols\clr.pdb\EDFF900AC9B94C1D9B32696A7759891A2\clr.pdb
    Loaded symbol image file: clr.dll
    Image path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
    Image name: clr.dll
    Timestamp:        Sun Apr 21 03:36:04 2013 (5173C114)
    CheckSum:         0095F379
    ImageSize:        00960000
    File version:     <b>4.0.30319.18052</b>
    Product version:  <b>4.0.30319.18052</b>
    File flags:       8 (Mask 3F) Private
    File OS:          4 Unknown Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft® .NET Framework
    InternalName:     clr.dll
    OriginalFilename: clr.dll
    ProductVersion:   <b>4.0.30319.18047</b>
    FileVersion:      <b>4.0.30319.18047</b> built by: FX45RTMGDR
    PrivateBuild:     DDBLD320
    FileDescription:  Microsoft .NET Runtime Common Language Runtime - WorkStation
    LegalCopyright:   © Microsoft Corporation.  All rights reserved.
    Comments:         Flavor=Retail
</pre>


Backing Details
---------------
The file Timestamp (0x5173C114), Checksum (0x0095F379), and Version (4.0.30319.18052) stored in the [MINIDUMP_MODULE][1] structure in the minidump's module-list-stream was for the newer CLR. Cracking open the minidump file myself and looking directly at the stream data:

<pre>MINIDUMP_MODULE : (pack:8 size:112) 
  +0x000 .BaseOfImage UInt64 : 8791579230208 (0x7FEF2F50000)
  +0x008 .SizeOfImage UInt32 : 9830400 (0x960000)
  +0x00C <b>.CheckSum UInt32 : 9827193 (0x95F379)</b>
  +0x010 <b>.TimeDateStamp UInt32 : 1366540564 (0x5173C114)</b>
  +0x014 .ModuleNameRva UInt32 : 107828 (0x1A534)
  +0x018 .VersionInfo tagVS_FIXEDFILEINFO : (pack:8 size:52) 
    +0x000 .dwSignature UInt32 : 4277077181 (0xFEEF04BD)
    +0x004 .dwStrucVersion UInt32 : 65536 (0x10000)
    +0x008 <b>.dwFileVersionMS UInt32 : 262144 (0x40000)</b>
    +0x00C <b>.dwFileVersionLS UInt32 : 1987004036 (0x766F4684)</b>
</pre>
Splitting the high and low words out of dwFileVersionMS we get 4 and 0.<br>
Splitting the high and low words out of dwFileVersionLS we get 30319 and 18052.

<br>
Using [dumpchk.exe][2], and looking at the module details in the PEB, we can see a different Timestamp (0x515530CE), one that actually corresponds to the older (18047) version:

<pre>7fef2f50000 <b>515530ce</b> Mar 28 23:12:30 2013 C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll</pre>

<br>
Looking at the raw memory in the crash dump where clr.dll is loaded, you can see the Checksum (0x00965F80) and Timestamp (0x515530CE) of version 4.0.30319.18047:

<pre>0:011> db 000007fe`f2f50000 
000007fe`f2f50000  4d 5a 90 00 03 00 00 00-04 00 00 00 ff ff 00 00  MZ..............
000007fe`f2f50010  b8 00 00 00 00 00 00 00-40 00 00 00 00 00 00 00  ........@.......
000007fe`f2f50020  00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00  ................
000007fe`f2f50030  00 00 00 00 00 00 00 00-00 00 00 00 18 01 00 00  ................
000007fe`f2f50040  0e 1f ba 0e 00 b4 09 cd-21 b8 01 4c cd 21 54 68  ........!..L.!Th
000007fe`f2f50050  69 73 20 70 72 6f 67 72-61 6d 20 63 61 6e 6e 6f  is program canno
000007fe`f2f50060  74 20 62 65 20 72 75 6e-20 69 6e 20 44 4f 53 20  t be run in DOS 
000007fe`f2f50070  6d 6f 64 65 2e 0d 0d 0a-24 00 00 00 00 00 00 00  mode....$.......
0:011> db
000007fe`f2f50080  39 e4 28 ed 7d 85 46 be-7d 85 46 be 7d 85 46 be  9.(.}.F.}.F.}.F.
000007fe`f2f50090  81 f2 f8 be 79 85 46 be-81 f2 fa be 74 85 46 be  ....y.F.....t.F.
000007fe`f2f500a0  74 fd c5 be 73 85 46 be-74 fd c2 be c9 85 46 be  t...s.F.t.....F.
000007fe`f2f500b0  ee 41 8d be 7f 85 46 be-e3 25 81 be 7c 85 46 be  .A....F..%..|.F.
000007fe`f2f500c0  ee 41 88 be 6b 85 46 be-ee 41 89 be 78 85 46 be  .A..k.F..A..x.F.
000007fe`f2f500d0  ee 41 8b be 64 85 46 be-7d 85 47 be ca 87 46 be  .A..d.F.}.G...F.
000007fe`f2f500e0  81 f2 ff be 76 85 46 be-ee 41 9e be 70 87 46 be  ....v.F..A..p.F.
000007fe`f2f500f0  ee 41 8c be 7c 85 46 be-ee 41 8f be 7c 85 46 be  .A..|.F..A..|.F.
0:011> 
000007fe`f2f50100  ee 41 8a be 7c 85 46 be-52 69 63 68 7d 85 46 be  .A..|.F.Rich}.F.
000007fe`f2f50110  00 00 00 00 00 00 00 00-50 45 00 00 64 86 06 00  ........PE..d...
000007fe`f2f50120  <b><i>ce 30 55 51</i></b> 00 00 00 00-00 00 00 00 f0 00 22 20  .0UQ.........." 
000007fe`f2f50130  0b 02 0b 00 00 90 69 00-00 c2 2b 00 00 00 00 00  ......i...+.....
000007fe`f2f50140  40 51 13 00 00 10 00 00-00 00 f5 f2 fe 07 00 00  @Q..............
000007fe`f2f50150  00 10 00 00 00 02 00 00-06 00 00 00 0a 00 00 00  ................
000007fe`f2f50160  06 00 00 00 00 00 00 00-00 e0 95 00 00 04 00 00  ................
000007fe`f2f50170  <b><i>80 5f 96 00</i></b> 02 00 60 01-00 00 10 00 00 00 00 00  ._....`.........
</pre>

I also jumped ahead in memory and looked at the Version resource in memory and saw the 18047 version string.

<br>
So now we have a minidump with conflicting information about what version of clr.dll was actually in use.

What Caused It
--------------
I also found out that our IT department recently pushed out a handful of Windows Updates, so:

 - While an application was running, an update to the CLR was installed.
 - The file in C:\Windows\Microsoft.NET\Framework64\v4.0.30319\ was updated to the newer version (4.0.30319.18052)
 - The application Crashed
 - MiniDumpWriteDump apparently used the file-on-disk information when storing the module list in the crash dump file (4.0.30319.18052)
 - WinDbg wasn't able to correlate the version stamped in the crash dump with what was in the process' memory as it had conflicting information.

To verify this, I manually modified the MINIDUMP_MODULE entry for clr.dll to change the Checksum, Timestamp, and Version from 18052 to 18047. After reloading the hacked up .dmp file in WinDbg and setting the exepath to the proper sos+clr dlls, I was able to successfully execute the sos commands and get a valid stack trace.

Bottom Line
===========
We essentially ended up with a minidump file that has conflicting information about which version of clr.dll is loaded into the process, preventing SOS from identifying the correct clr engine.
This is likely a bug in MiniDumpWriteDump as it saves out the crash dump file. But should only happen when the backing version of the CLR has been updated while your application was running.


  [1]: http://msdn.microsoft.com/en-us/library/windows/desktop/ms680392%28v=vs.85%29.aspx
  [2]: http://msdn.microsoft.com/en-us/library/windows/hardware/ff560109%28v=vs.85%29.aspx
  • 1
    You post is indeed fine; it's not formatted as code because it shouldn't be formatted as code; using <pre> is the appropriate markup to use here. – Servy Aug 13 '13 at 16:53
5

The algorithm that looks for what it thinks is code not properly formatted as code ought to ignore any content inside of <pre> tags.

As a workaround, until this is (hopefully) changed, you can format those blocks as code to submit the post, and then remove the code formatting in an edit. As much as it pains me to mention this exploit; since it is indeed the filter that is incorrect, not you, this is one place in which it would result in the proper actions taken.

| improve this answer | |
  • 1
    The lang-none hint may have something to do with it as well. If you are going to use the language hint you may as well indent the code instead of using <pre> tags. The hint also needs to reside just above the indented content to which it applies. – BoltClock's a Unicorn Aug 13 '13 at 17:02
  • @BoltClock'saUnicorn - I added the lang-none hint to try and work around the rejection. It obviously didn't work, but I decided to leave it in to make my intent clear (that there shouldn't be any language related formatting applied to anything). – josh poley Aug 13 '13 at 17:05
  • 1
    Oh, I see the <pre> tags are there to allow the HTML within the samples to work. That complicates things... – BoltClock's a Unicorn Aug 13 '13 at 17:13
  • @Servy - They have apparently "fixed" this loophole. I posted the answer converting the pre's to happy code blocks. But I'm unable to change them back during an Edit (get the same error when try to submit the edit). I also tried to removed the language hint, to no avail. – josh poley Aug 13 '13 at 17:15
  • 1
    @joshpoley Apparently the restriction doesn't apply to users with as much rep as I have. I've edited in the <pre> tags for you. – Servy Aug 13 '13 at 17:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .