Saturday, February 07, 2009

Custom Win PE Boot Disk Building: Step Four – Pulling it all together.

Hang on. This may be a doozie.

It will (hopefully) wrap up this project.

Summary

The purpose of this ongoing project is to build a Win PE 2.0 based boot-disk, that has a great VistaPE GUI interface (instead of the standard CLI shell) and the PGP WDE drivers injected so we can “liveCD-boot” a PGP WDE system (assuming we have the user’s passphrase).  Oh yes, and it has to handle the Dell GX 7xx series USB keyboard drivers.

If you are just joining us, please go back and review the following posts to get up to speed:

Note: I’m serious about this, if you haven’t read and stepped through the previous posts there is a good chance you will be lost.  Also, this post assumes you are familiar with working with Microsoft’s ImageX WIM files; mounting them, committing changes to them, etc.  If not then there is a good chance I will loose you here.

Mkay?

Onward!

When last we left our intrepid hero…

At the end of the last post, I was poking around in the VistaPE WAIK build WIM file and looking at how it worked.

I had started with the winpeshl.ini file that loads the VistaPE project’s shell replacement.

That was handled by the custom executable vpeldr.exe and it’s configuration file vistape.cfg.

Turns out that the VistaPE project developer, NightMan, has already provided us with all the documentation we need to understand just how that miracle functions:

It’s a good read and I really encourage everyone to take the time to read and understand it. It really is an amazing piece of work from NightMan and he should be fairly recognized for it.

When I opened up my particular vistape.cfg file to review, I found it contained almost all the information I needed to trace out, extract, and then inject the “ripped” contents into my already PGP Injected Win PE 2.0 wim.

Sweet.

WIM Mounting

I’m sure if you are still reading this post, you have already configured your system to allow for the mounting of ImageX wim files.  If not, please take a moment to read this earlier post:

Yes, you can use ImageX to mount your wim files via the CLI, but with all the work we have done and have yet to do, it seems much more efficient for me to work with them from a GUI interface.

For the longest time I preferred the method of adding this quick-mount feature to the Windows right-click shell context: Mounting WIM Images from Windows Explorer - Robert McLaws: Windows Vista Edition

However, I would occasionally get lost with my mount/unmount points and wims in progress and would trip myself up.

Then I graduated up to the ImageX GUI (GImageX) coded by Jonathan Bennett.

The latest version is v2.0.14 released in October 2008 and it is sharp.  He actually now has a beta version that supports the upgraded version of ImageX found in the Windows 7 Beta WAIK.

So either way, be prepared.  I’m using the GImageX to do my work here.

Bonus Tip:  I’m also using the freeware dual-pane/tabbed window explorer alternative FreeCommander not just because I love it but the dual-pane window makes file-copy actions like what we are about to do so much easier than Windows Explorer.  There are a lot of other great alternative file managers as well you might want to look into trying.  Your call…

Organ Harvesting

I created an empty folder at the root of C: called “mounted_wim” to use as my mounting point for wim files.  That was a carryover from my foray into using Robert McLaw’s mounting solution.  I’ve kept with it.

I also created another empty folder at the root of C: called “extracted_items”.  This is where I am going to temporarily place the files/folders I pull out of the VistaPE WAIK project that I need.  You might want to go ahead and make a “system32” folder in there as well while you are at it.

Using whatever your preferred method of wim mounting is, go ahead and mount up the VistaPE wim you created in our Custom Win PE Boot Disk Building: Step Three – VistaPE 12 RC1 Walkthrough step.

If you followed those steps, you should find it in the following location:

C:\VistaPE_WinBuilder_v12RC1\Target\VistaPE-Core\vistape.wim

OK?

Browse to the location you mounted it and copy the following items into your “extracted_items” folder.

Depending on how you roll, you might want to make some notes on a scratch-pad of where stuff came from so you can put it back correctly.

From the VistaPE.wim file we want to copy:

  • “Program Files” <—the folder and all its sub-contents.  Note I just copy this folder as is into the extracted_items folder.

Now browse deeper into the windows\system32 folder and locate and copy the following items into your “C:\extracted_items\system32” folder:

  • devcon.exe
  • hw.bat
  • HWPnp.exe
  • HWPnP.htm
  • HWPnPDLL.dll
  • vistape.cfg
  • vpeldr.exe
  • winpe.bmp
  • winpeshl.ini

Want to explore this particular WIM a moment since we are here?  Fine.  Just don’t take any more relics this time round.

All done?

Unmount your vistape.wim file from your system.

Stage 1: Transplant the Brains

Now go and this time mount up our previously PGP WDE driver injected Win PE 2.0 WIM file.

C:\winpe_x86\ISO\sources\boot.wim

This time, be sure to mount it “Read and Write” so we can actually make changes to it.  (Note: you might want to make a copy of the working original first, just in case you make a mistake!)

Into the C:\mounted_wim\Program Files folder, let’s copy back the sub-folder contents from our C:\extracted_items\Program Files location.

Got em all tucked in?  Great!

Now browse to your C:\extracted_items\system32 and copy all those items into the C:\mounted_wim\Windows\System32 folder.

You are doing great!

Tweak the HWPnP module

Now, in previously looking at the vistape.cfg file, I located that it called to a HWPnP.exe file that uses the hw.bat file to supplement it.  Research on that file indicates it is a Plug-n-Play helper used in BartPE and VistaPE projects.

The VistaPE WAIK build shows the second line the hw.bat file acts off of is as follows:

HWPnP.exe +all -storage\volume -USB\ROOT_HUB +USB\ /log /p /u /d+ /a

Since I knew the Dell USB keyboard hub controller driver was giving me fits, I decided to err on the side of caution and disable this.  This may not actually be needed.  It’s up to you.

Open up the hw.bat file and add two colons in front of it to REM it out thusly

::HWPnP.exe +all -storage\volume -USB\ROOT_HUB +USB\ /log /p /u /d+ /a

Save the modified file back.

Dismount that wim and be sure to use the option to “commit changes” so our changes actually get written and applied to the winpe.wim file.  This part might take a while to process as the changes are written and the wim file is updated.

Stage 2: Transplant the Lungs

Now, we know from our l33t Win PE 2.0 hacking skillz that any files/folders we add into the C:\winpe_x86\ISO folder will be included on the root of our final boot disk.  Right?

So lets work on that next.

Browse back to 

C:\VistaPE_WinBuilder_v12RC1\Target\VistaPE-Core

In there, copy the following items:

  • “Programs” <—the folder and all its sub-contents.
  • vistape.cfg

And paste them into the following folder.

C:\winpe_x86\ISO

I always be sure I have a copy of the ImageX.exe file copied in there as well.  That’s up to you.  It makes system imaging and capture so easy.

You can add in other things as well to that folder (once you figure out how much extra space you have) such as portable system utilities and other stuff. Just don’t add anything that modifies or overwrites the things we have already added in there.  I prefer to make another subfolder called “utilities” and dump all my special stuff in there just to be safe.

Stage 3-option 1: Sewing it Up (for standard CD disk media)

Go to the Start menu and under All Programs find the Microsoft Windows AIK folder and launch Windows PE Tools Command Prompt, or open a command prompt and type

cd c:\program files\Windows AIK\Tools\PETools.

Then, type

oscdimg -n -bc:\winpe_x86\etfsboot.com c:\winpe_x86\ISO c:\winpe_x86\winpe_x86.iso

(all one line)

An ISO file will be created inside the c:\winpe_86 folder.

With the ISO image file created, you can now burn the image file to CD.

Stage 3-option 2: Sewing it Up (for standard DVD disk media)

I personally stick with the CD sized builds as almost every system we touch (old and new) comes with a CD-ROM drive.  Newer systems have DVD-ROM drives, but if I go to service an older system the DVD disk isn’t going to help me.

If you try to burn a DVD sized ISO with the normal instructions above it will error out.  You must use the undocumented "-m" argument in your string to force it to build the ISO over the normal CD-sized ISO size.  Use the -m switch to override the creation of ISO images larger than 700 MB

In that case (because you got all crazy with adding lots of extra custom utilities and stuff into your C:\winpe_x86\ISO folder) type

oscdimg -n -h -m -bc:\winpe_x86\etfsboot.com c:\winpe_x86\ISO c:\winpe_x86\winpe_x86.iso

(again, all one line).

With the image file created, you can now burn it to a DVD disk. (assuming that the combined file-size isn't larger than the DVD's storage capacity.)

Testing

I ALWAYS pre-test my built boot ISO files in a Virtual PC 2007 session first before burning.  That way if I did something wrong I will see it before making coasters.

If all goes well you should see the following (click for a slightly larger view and yes, I know the wallpaper may be a bit different on yours…read on for how to customize that).  Note the PGP drivers are working (even though the virtual drive I am using isn’t actually PGP WDE encrypted…).

image

If it looks good and seems to work cleanly, burn your custom “Win PE 2.0 + PGP WDE driver injected + VistaPE WAIK build ripped + Dell Optiplex USB keyboard driver working” Frankenstein’ish ISO (located as C:\winpe_x86\winpe_x86.iso) to a disk and see what happens on your live systems!

Caveats

Ripping out the core elements of Vista PE WAIK build as I have documented works.  But it might make NightMan and other hard-core VistaPE/WinBuilder pro’s groan.  I’m also likely leaving some critical bits behind.

So there are some applications that would work fine in traditional VistaPE build that error out and will not run under this GSD project.  Be aware of that.

Second, if you go back in and mount the “C:\winpe_x86\ISO\sources\boot.wim” file, make more changes (wallpaper below for example) and re-commit the changes, you will likely notice your boot.wim file size continues to grow.   This is not just because you added stuff, even if you “delete” stuff from your mounted/committed wim, it may still grow.  That’s because with this method of wim management the changes haven’t been optimized to the wim.  You need to actually export your wim file in a different ImageX process to reclaim the space when you have modified it.  See this post (and the Google) for more information on that.  I normally don’t bother as my final ISO size is still under the limits for a CD burn.

Also, if you compare the vistape.wim used to boot a standard VistaPE WAIK project to the one we did using a (PGP modified) base Win PE 2.0 boot wim you will see close to the following:

  • C:\VistaPE_WinBuilder_v12RC1\Target\VistaPE-Core\vistape.wim  Size = 102.86 MB
  • C:\winpe_x86\ISO\sources\boot.wim Size = 187.11 MB

NightMan and all have worked hard on the original VistaPE WinBuilder project to remove many files/folders and other items from the standard Win PE 2.0 WAIK wim file that are not necessary or needed to work in his project.  That’s why the size is so much smaller.

But in doing so, something caused the Dell Optiplex 7xx “USB Keyboard Hub” and related HID drivers from loading.

This method works around that.

I haven’t tested it on other systems, just those that I and my team service and support.  It might work just fine on other hardware system configurations with that same issue.  It might not.  I don’t know. 

If you take the time to do all this and find it does help you load previously unavailable USB keyboard devices on your own particular system, please leave a note in the comments so others (and myself) will learn of your success.

As always, what works for me and our systems, may not for you.  Your mileage may vary.

Also, I’ve got at least one more related post in this series planned to address coming developments I have found with future VistaPE/WinBuilder projects as well as Win PE 3.0 in general.

So stay tuned for that.

Optional Tweaks and Tips

As I mentioned before, if you want to kick things up a notch, add in some additional portable applications, and fire them up off via PStart you can add a modified version of this to your startnet.cmd file before you pack up your customized wim: (via RunOnceEx from CD by Alanoll)

wpeinit

cmd.exe

For %%i IN (D E F G H I J K) DO IF EXIST %%i:\cd-specific_filename SET CDROM=%%i:

%CDROM%\PSTART\PSTART.EXE

Also, if you want to add a sexy custom desktop wallpaper (please comply with any work-rules if you distribute such builds in your workplace and use something tamer) just find your image (I prefer to use 1024x786 sized), and convert it to a BMP format file and save with the name “winpe.bmp” and place back in your custom wim’s windows\system32 folder overwriting the original one we extracted from the VistaPE WAIK built wim.

I have also discovered that if you want to plug in a USB storage device to your Win PE booted system it doesn’t always pick up the new drive.

Open a cmd session and run the command “DiskPart” then do a “list disk”,

That is usually sufficient for the OS to scan for any drives and mount what it detects.

Refresh your windows explorer tool (in this build it would be Cubic Explorer) and you should see it now just fine.

Tomb Raiding using the PGP WDE drivers and pgpwde.exe

Remember, one of the other reasons I needed to go through all this pain and learning was to be able to boot one of our PGP WDE encrypted systems “off-line” with a VistaPE’ish Win PE boot disk and use the PGP injected drivers to access the “on-the-fly” decrypted contents of the drive rather than taking hours (or more) to do a full off-line drive decryption using the stock PGP WDE recovery boot-disk (and see this PGP link on how to use), before being able to access the contents with a traditional boot-cd.

Our hard work and dedication has now given us a much more flexible and powerful tool!

Boot the PGP WDE system with our custom boot disk.

Of course, you will need to know the user’s PGP WDE passphrase for on-the-fly file/system access and recovery to work….

Open up a CMD session.

The basic commands I use are as follows

  • pgpwde –enum
  • pgpwde --disk 0 –status
  • pgpwde --disk 0 --auth -p “xxxx”   (NOTE: put passphrase in “  “ if separated with spaces.)
  • pgpwde -h

The first one “enumerates” the system disks and volumes.

The second one shows the status of any pending WDE encryption/decryption activities.  This might tell you if WDE was initialized and is x% completed.

The third command is the money-shot.  This is what our tomb-raiding has been working towards. It allows you (with the user’s PGP WDE passphase) to decrypt the drives “on-the-fly” from the Win PE 2.0 environment so you can recover the files from a non-bootable system if the OS has crashed (or for other reasons).  The trick to this one is that if the user’s PGP WDE passphrase has spaces in it, you must enclose it all in quotations marks.

I haven’t tested to see if this will take a PGP WDE recovery token and if so, what impact that might have.  In our cases (so far) the user is present and can provide us with their passphrase.  If they have disappeared and their passphrase is not available, then you would need to try a PGP WDE recovery token instead.  I’ll let you know if I try it and it works.

There are many, many more powerful PGP commands and arguments for the pgpwde.exe executable. Do the fourth command to list them all.  Then search the Google for some powerful and undocumented ones as well.

Either on a live PGP booted system or a LiveCD PGP injected system, the pgpwde.exe command line tools are very good to know and be familiar with if you support such systems.

Final Warning: You must inject and use the PGP WDE drivers/tools specific to the version of PGP WDE deployed on your system(s).  Failure to do so might seriously muck things up!  Mkay?

Whew!

Thanks for sticking with me.  I hope this helps someone (or two).

I also suppose if you had the VistaPE USB system keyboard sensing issue but didn’t need the PGP WDE drivers, you could just skip over that step and not do that.  The rest should work fine.

Wicked Cool and Sexy, isn’t it?

Just like I promised….

Cheers!

--Claus V.

25 comments:

Nathaniel said...

Finally got around to reading this. You're pretty determined, Claus. Nice job. Closest thing I've done is build a BartPE USB that just goes in a reboot circle (so useful).

Anonymous said...

In the Step 3's you may want to fix some of the paths.

{arch}=x86, ia64, amd64 as determined by platform

..\{arch}\oscdimg -n -bc:\winpe_x86\etfsboot.com c:\winpe_x86\ISO c:\winpe_x86\winpe_x86.iso

I love that you've done all this incredible work.

Keep it up!

Pete said...

Very cool - Using VPELDR without the hardware detect speeds up the boot by 50%, and still lets you run the BSexplorer.

This has helped me tons, Thank you very much.

Ryan said...

Claus-

Didn't know where to post this. I read somewhere in all your posts that your techs use CubicExplorer and you are now building PE images on 3.0. I am having problems making CubicExplorer work even with a simple vanilla PE 3.0 setup. ...any pointers?

Ryan-

Claus said...

@ Ryan -- Hmmm. Good question. Yes. The techs continue to use CubicExplorer just fine on my custom PE builds. It worked under the original standard VistaPE project build on Win PE 2.0 (Vista). Then I found it seemed to work just fine under the beta/RC builds of the Win 7 WAIK releases (WinPE 3.0) that I passed out a while back and they continue to use almost daily.

Curiously, just last weekend I decided for kicks-n-grins to make a fresh bootable USB stick for my home needs using the steps in this post (sans the PGP WDE shims) and the final release Win7 WAIK with Win PE 3.0. Cubic Explorer continues to work absolutely fine with this final version.

I'll do some version checks between the CubicExplorer that I am using and what version is on the Net. AFAIK I'm using the same version I snagged in my original VistaPE 2.0 building days so it might be older (and more compatible?) than a newer release (if such a thing exists). I'll also take a look in the custom files that I'm copying over in this walkthough that pertain specicially to Cubic Explorer as best I can find.

Maybe something in the version?

While it shouldn't have anything to do with rights issues on the files, I had to give my original VistaPE build project folder (and sub-contents by inheritance) "Everyone" rights to build correctly.

While I think that PE 3.0 runs with "System" rights, could it be a rights issues on some of the Cubic Explorer files/folders?

What kind of error is it throwing? Or is it just not executing at all?

Can you not launch it from the desktop?

What about if you browse/launch the executable directly from a CMD session?

Stay tuned and I'll drop another comment here with what I find out.

To be honest, while I prefer myself to generally zip around in the CMD now, there are some things that a GUI file manager just makes so much easier. When I do need one, I just browse to where I stashed the "portable" program folder for FreeCommander and use it. Both it and A43 seem to work just fine for me under PE 3.0 as well. Don't limit yourself just to CubicExplorer as there are many other great alternatives as well. (Actually, I have my home-brew WinPE 3.0 custom build set to kick off PStart once it loads all the way up, and then I had configured PStart to link to all my on-board USB "portable" tools and utilities, of which FreeCommander and A43 are among the listed...I guess that's another post needed as well....)

I left CubicExplorer on my WinPE builds as an easy carryover from the VistaPE project build I raided to start building my custom ones.

As I'm learning more about the "custom" structure in these PE hot-rod builds, I'm finding it a bit easier to tweak with different apps that I like a bit more, personally.

Anyway....I'll let you know what I uncover about CubicExplorer specifically.

Thanks for asking!

--Claus V.

Claus said...

@ Ryan -- I've have a few moments to look at my CE build which is working under the various Win PE 3.0 builds I have used (beta/RC/final).

It lives in a folder called "CubicExplorer" tucked in the "Program Files" folder which is then added to the PE wim file accordingly.

I only found one other reference to it elsewhere in the custom files. That was in the vistape.cfg file taken out of/stored in the Windows\system32 folder as noted in the post. see the end of this one for the contents on my systems.

The version that I found we are using is dated 8/8/08 and reports in as file version 0.90.0.1084, product version 0.90 RC 3.

Are you trying to use a more recent version of CE? If so this much older version may not be requiring some dependencies that newer ones could be.

Hope this helps.

--Claus V.

Text contents from my vistape.cfg below:

[main]
screen=1024x768
defaultshell=BS Explorer

[shortcut]
System|DOSShell|%SystemDrive%\Program Files\DOSShell\DOSShell.exe||
Security|DiskCryptor|%SystemDrive%\Program Files\DiskCryptor\dcrypt.exe||
Office|AngelWriter|%SystemDrive%\Program Files\AngelWriter\AngelWriter.exe||
Network|Miranda@HotCoffee|%SystemDrive%\Program Files\Miranda-HotCoffee\HotCoffee.exe||
Network|LanSpy|%SystemDrive%\Program Files\LanSpy\LanSpy.exe||
Network|IP-Tools|%SystemDrive%\Program Files\IP-Tools\IP_Tools.exe||
|Far|%SystemDrive%\Program Files\Far\far.exe||
Desktop|CubicExplorer|%SystemDrive%\Program Files\CubicExplorer\CubicExplorer.exe||
|CubicExplorer|%SystemDrive%\Program Files\CubicExplorer\CubicExplorer.exe||
Disk Tools|Victoria 4.3|%SystemDrive%\Program Files\vcr43\victoria43.exe||
System Information|System Info|%SystemDrive%\Program Files\System Info\siw.exe||
System Information|Unknown Devices|%SystemDrive%\Program Files\UnknownDevices\UnknownDevices.exe||
QuickLaunch|Total Commander|%SystemDrive%\Program Files\totalcmd\totalcmd.exe||
Desktop|Total Commander|%SystemDrive%\Program Files\totalcmd\totalcmd.exe||
|Total Commander|%SystemDrive%\Program Files\totalcmd\totalcmd.exe||
System|Starfield ScreenSaver|%SystemRoot%\System32\Starfield.scr||
System|Ruslat|%SystemRoot%\System32\ruslat.exe||
Network|Map Network Drive|%SystemDrive%\Program Files\MapDrive\MapNetworkDrive.exe||
Disk Tools|Disk Partitioner|%SystemDrive%\Program Files\DiskPartitioner\DiskPartitioner.exe||
System|Device Manager|%SystemRoot%\system32\DevMan.exe||

[autorun]
nowait|Ruslat|%SystemRoot%\System32\ruslat.exe

[registry]

[association]

[shells]
Far|Far\far.exe
CubicExplorer|CubicExplorer\CubicExplorer.exe
Total Commander|totalcmd\totalcmd.exe
BS Explorer|BSExplorer\Explorer.exe

[postconfig]
hidewait|Human Interface Device Access|x:\windows\system32\net.exe start "hidserv"
hidewait|Computer Browser|x:\windows\system32\net.exe start "Browser"
hidewait|User Profile Service|x:\windows\system32\net.exe start "ProfSvc"
hidewait|Server|x:\windows\system32\net.exe start "LanmanServer"
hidewait|DNS Client|x:\windows\system32\net.exe start "Dnscache"

Ryan said...

POST PART A...

@ Claus - Sorry for my delay, I am Internet impaired (is that a disability?) right now which means I am moving as fast as the Slowskys (Comcast Turtles). For some background, I have been using WinPE for a very long time. I created an imaging solution for WinPE 1.0 w/ a Bluebox shell using a severely hacked (think DIY Dentistry pain!) NetWare Client32 and xcopy with a HTA front end. It was pretty, but it was slow as hell! ...what is it with me and slow today.

Let's shift this puppy into a higher gear! To take the focus off of any mods I made to PE3, I built a plain 'ol vanilla one from WAIK 3.0 RTM with this script.

//VANILLA PE3 BUILD
Dism /Mount-Wim /WimFile:c:\winpe_x86\winpe.wim /index:1 /MountDir:c:\winpe_x86\mount
Dism /image:c:\winpe_x86\mount /Add-Package /PackagePath:"C:\Program Files\Windows AIK\Tools\PETools\x86\WinPE_FPs\winpe-hta.cab"
Dism /image:c:\winpe_x86\mount /Add-Package /PackagePath:"C:\Program Files\Windows AIK\Tools\PETools\x86\WinPE_FPs\en-us\winpe-hta_en-us.cab"
Dism /image:c:\winpe_x86\mount /Add-Package /PackagePath:"C:\Program Files\Windows AIK\Tools\PETools\x86\WinPE_FPs\winpe-legacysetup.cab"
Dism /image:c:\winpe_x86\mount /Add-Package /PackagePath:"C:\Program Files\Windows AIK\Tools\PETools\x86\WinPE_FPs\en-us\winpe-legacysetup_en-us.cab"
Dism /image:c:\winpe_x86\mount /Add-Package /PackagePath:"C:\Program Files\Windows AIK\Tools\PETools\x86\WinPE_FPs\winpe-mdac.cab"
Dism /image:c:\winpe_x86\mount /Add-Package /PackagePath:"C:\Program Files\Windows AIK\Tools\PETools\x86\WinPE_FPs\en-us\winpe-mdac_en-us.cab"
Dism /image:c:\winpe_x86\mount /Add-Package /PackagePath:"C:\Program Files\Windows AIK\Tools\PETools\x86\WinPE_FPs\winpe-pppoe.cab"
Dism /image:c:\winpe_x86\mount /Add-Package /PackagePath:"C:\Program Files\Windows AIK\Tools\PETools\x86\WinPE_FPs\en-us\winpe-PPPOE_en-us.cab"
Dism /image:c:\winpe_x86\mount /Add-Package /PackagePath:"C:\Program Files\Windows AIK\Tools\PETools\x86\WinPE_FPs\winpe-scripting.cab"
Dism /image:c:\winpe_x86\mount /Add-Package /PackagePath:"C:\Program Files\Windows AIK\Tools\PETools\x86\WinPE_FPs\en-us\winpe-scripting_en-us.cab"
Dism /image:c:\winpe_x86\mount /Add-Package /PackagePath:"C:\Program Files\Windows AIK\Tools\PETools\x86\WinPE_FPs\winpe-wmi.cab"
Dism /image:c:\winpe_x86\mount /Add-Package /PackagePath:"C:\Program Files\Windows AIK\Tools\PETools\x86\WinPE_FPs\en-us\winpe-wmi_en-us.cab"

Ryan said...

Meaning of Life, Part B...

Crap, crap, crappity-crap crap! Your form is limiting me to 4096 characters but I have more to post. This is the top few lines of the error report that CE kicks. I am running it from the desktop with Everyone/Full Control. You will see I am running v0.90.0.1131 which is the website says is the final. How do I get an old (better?) version to try? Thought... Not enough scratch space?

//CE NASTYGRAM
date/time : 2009-11-02, 13:27:47, 419ms
operating system : Windows NT New build 7600
system language : English
program up time : 3 seconds
processors : 2x Intel(R) Atom(TM) CPU N270 @ 1.60GHz
physical memory : 1704/2038 MB (free/total)
free disk space : (X:) 31.06 MB (C:) 11.23 GB
display mode : 800x600, 32 bit
process id : $4f8
allocated memory : 16.00 MB
executable : CubicExplorer.exe
exec. date/time : 2009-04-03 17:43
version : 0.90.0.1131
madExcept version : 3.0i beta 2
callstack crc : $950ceb1b, $45f1f642, $cc0df575
count : 5
exception number : 1
exception class : EAccessViolation
exception message : Access violation at address 00626175 in module 'CubicExplorer.exe'. Read of address 00000004.

Claus said...

@ Ryan - No worries! Sorry about the Blogger comment limits. Maybe one day I will port this blog over to a hosted solution...

I've uploaded a 7zip file of my own Cubic Explorer build from yore.

I've temporarily uploaded it to my box.net account. Use the following link to grab it.
http://www.box.net/shared/o59qk2h66p

Let me know once you've got it so I can drop if after a few days.

See if that one works better.

To be honest I've not spent hardly any time with the DISM and script building. I just did my basic VistaPE (WinBuilder) work and then as listed in the walkthrough, stripped out the files/directories and ported them into a standard basic WAIK build of PE 2.0 / 3.0.

Maybe I'm just darn lucky...

Your memory settings look great so I'll download the latest CubicExplorer and drop it onto my USB bootable stick (PE 3.0 now) and see what happens when I try to run the same one you have been using.

Let me know what you find out.

Cheers!

Claus V.

Claus said...

@ Ryan - as a followup, I downloaded and dropped the very latest Cubic Explorer release onto my 16 GB WinPE 3.0 flash drive build.

Granted, I didn't "integrate" it into the PE build and shell, but only dumped it off the drive root for testing purposes.

I browsed and launched it in the PE 3.0 environment just fine and I didn't have any errors. The older "integrated" version worked fine as well. So I am a bit stumped.

I suppose if you don't have much luck (and my free box.net account supports the size), I could pack up all the extra PE files I mention in my custom building and drop them for you to pull and test with your base boot.wim file.

I wouldn't include that one, so you would have to do my steps and "integrate" mine into your mounted wim file, but it might be helpful to work with for testing and comparison purposes...

Let me know and I can probably try to get things packed up over the weekend and uploaded.

If box.net size limits don't work and you are still interested, I'll see about an alternative upload solution.

Cheers.

Claus V.

Ryan said...

@ Claus

1. Downloaded CEb3 - you can kill the download.

2. CEb3 did the same thing.

3. FreeCommander works, but I am more concerned at the underlying reason WHY CE is failing. ...what else might fail?

4. I don't want CE for myself, but for other techs that whine that they want explorer. CE was the closest thing I have seen that "looks" like explorer. They only need the ability to copy/paste.

5. Can you verify that I am not missing something fundamental in my build script from post 7 - Vanilla PE3 Build. I made it easy if you want to test it... http://www.box.net/shared/or7445js8u It is just a batch file to build the same version I am using. Check it out first just to make sure you know what it is doing. (duh) I don't want to be blamed for formatting your C: drive. ;^)

6. I was talking to John Baker (Microsoft IT Pro Evangelist - http://blogs.technet.com/johnbaker ) yesterday about some issues I am having. He said I might want to check out MDT 2010 which RTM'd about 5 weeks ago. He said he thinks it includes an updated version of WinPE 3.0. The download is <10Mb in size, so I find that hard to believe. But, you may want to check it out.

Ryan-

Claus said...

@ Ryan -- Thanks for the extra info!

1) Will do.

2 & 3) Definitely seems like something in the building is the cause. My PE 3.0 is handling the old and new CE builds just fine.

4) I fully understand! Today I PE booted, zeroed out, diskparted, formatted and re-imagex'd over 35 systems from the CLI in about 5 hours. My supporting techs just couldn't keep up with my CLI work and kept getting lost keeping track of the drive letter assignments, etc....

5) I'll take a close look at it this weekend. I downloaded the batch file right now. Looking forward to playing with it!

6) Found it. Microsoft Deployment Toolkit (MDT) 2010 Will be checking it out as well. I think the statement regarding the size was technically accurate but from reading the download page, other components may be needed as well (like the WAIK) so those would dramatically increase the effective size if selected. This page seemed to have some additional documentation helps: Now Available! Microsoft Deployment Toolkit 2010 I'm only deploying XP Pro systems at work but I'm also intrigued by tools that make the PE building process easier so I'll have to explore this as well based on your tip.

Stay tuned!

Cheers!
--Claus V.

Claus said...

@ Ryan -- Busy weekend but I've had a chance to take look at the bat file as well as review the prior information.

Question. Are you dropping the CE program into the WIM itself, then unmounting/committing it? Or is the CE application outside of the WIM but included as part of the folder structure that gets made into the ISO file?

I'm really having fun researching this! I hope we can figure it out.

I'm hoping to get a chance to respond either later this afternoon or by Sunday with some thoughts I am being led to..

Cheers!
Claus V.

Ryan said...

@ Claus

I have tried it both ways to no avail.

I downloaded the Win7 OPK (legally) to try this out on as well. Maybe one of my FPs is misbehaving from the AIK?

Ryan-

Claus said...

@ Ryan -- OK. Good info.

Stay tuned. On call this weekend and running in/out of the house for wife/daughter errands on top of that!

Promise to have some more tips to try for you...

;-)

Claus V.

Claus said...

@ Ryan -- I could be wrong here but after the starts and stops doing some research on your building technique and looking at your PE DISM bat file, I wonder if maybe you answered your own problem...

processors : 2x Intel(R) Atom(TM) CPU N270 @ 1.60GHz
physical memory : 1704/2038 MB (free/total)
free disk space : (X:) 31.06 MB (C:) 11.23 GB
exception class : EAccessViolation
exception message : Access violation at address 00626175 in module 'CubicExplorer.exe'. Read of address 00000004.

Your info shows a fairly beefy system with good CPU and System RAM.

I did see that the ramdisk is just 31.06 MB. Not very big. Couple that with the exception message seeming to be memory access related.

You mentioned "Thought... Not enough scratch space?"

I don't build with DSIM so I was trying to sort out all the modules you were loading via the DSIM bat file.

Found them described here in this technet info page: Building a Windows PE Image

So, it appears you are loading all of these (and I assume you actually need all these features in your PE disk...).

I just take the raw WAIK x86 Winpe.wim file and change it to boot.wim and then inject the specific VistaPE components mentioned in the post. I'm not adding any the optional packages you are. So that appears to be the biggest difference.

I also found reference to the WMI package soaking up Ramdisk space in some cases WMI Tracing.

So I am leaning to your own diagnosis of not having enough scratch space.

So how to fix?

Turns out (I learned this today) that by default, a Win PE 3.0 (I think 2.0 also) sets the scratch space (RAM disk) at 32 MB. That is close to what you captured.

In Win PE 2.0 you could use the PEImg tool to modify the scratch-space when building...PEImg Command-Line Options.

However that tool has been removed from the Win 7 WAIK PE kit.

Luckilly, DISM now takes that role and the method is very easily found in the very-same TechNet reference I first linked: Building a Windows PE Image at the very bottom.

Quoting:

"If your Windows PE environment becomes unresponsive when running an application, you may have run out of memory. By default, Windows PE allocates 32 megabytes (MB) of writeable memory, known as scratch space. You can increase the scratch space up to 512 MB by typing the following at a command prompt:

dism /image:mounted-image-path/Set-ScratchSpace:size

Where size can be any one of the following values: 32, 64, 128, 256 and 512."

End Quote.

So I would recommend trying to add the following line above to your DISM PE bat file and increase maybe to 64 or 128 and see if that helps.

If not, then try to build your PE image again via your PE bat file, but each time, just leave out one of the optional packages you are adding. Again, I'm not using them and my PE works great for CubicExplorer. Possibly between increasing the Scratch Space value and/or pulling out a package that is causing an application conflict you can resolve the issue.

I can't wait to try my own hand now at some DISM based building of PE packages as well as trying to see the benefit (if any) of increasing the scratch space on my own ones. This might improve the performance of my own custom PE builds as well.

Please let me know if you have any thoughts on this as well as any successes (or failures) this line of exploration brings.

If it is successful, I'd like to blog about it. I've got quite a few more great links out of this learning you've led me through!

Cheers!

--Claus V.

Anonymous said...

@Claus

I'm having the same issue as Ryan, with the latest version of CubicExplorer not working in WinPE 3 (acquired as part of the Windows 7 OPK).

I receive the same error message that Ryan does. I've tried increasing the scratch space to 512MB and it doesn't make any difference.

I'm using a "stock" boot.wim that is an exact copy of the winpe.wim that is included in the OPK. The only changes I'm making are adding CubicExplorer.exe (0.90.0.1131) and increasing the amount of scratch space. I've tried including the various .xml files, as well as "Locale" and "Skins" directories in the PE image, alongside the CE executable, but CE still crashes.

I'm surprised that it works on your PE 3 build. Are you using the boot.wim that's included with the latest version of the Windows 7 OPK?

Ryan said...

I too never overcame this obstacle.

Claus said...

@ Anonymous - Wow. That is really strange.

No. I am not using the WinPE boot.wim from the Windows 7 OEM Preinstallation Kit (OPK).

I'm using the Windows Automated Installation Kit (AIK) for Windows 7. I could be off but I think this is the final release version.

I can't honestly say what differences there might be in the stock boot.wim files between the two.

I've not had any issues using it with scratch-space set down from the default 32 MB to the top end 512 MB.

I nearing a loss to explain the crazy behavior that Ryan and you have reported. Might be time to toss on Process Monitor to find the fail point.

I do my building on NTFS partitions, and I seem to recall I set the security permissions on the building folders to "everyone" in the past due to that being a need back from the VistaPE project building work I did.

I can't imagine that it would be a security permissions issue causing the issues. I've not had any issues with it being built/run on my XP Pro, XP Home, or even my Vista Home or Windows 7 Home systems. Those are all final releases. It also worked on my Windows 7 RC build I used for a while.

This weekend will be a wash but maybe next weekend I can look at things again. Been a while since I have and the break might help.

Will post an update soon.

If you get any more clues please leave an update.

--Claus V.

Ryan Carver said...

I forgot to say that I have tried both the WAIK and OPK.

Claus said...

@ Ryan and Anonymous -- Woot! I've been able to finally generate a successful failure!

I got a base WAIK PE3.0 to fail finally with CubicExplorer.

I also got some great ProcessMonitor log captures of the events.

While I didn't catch anyone shooting JR just yet, I've got some suspicions. Now that I've got a ISO baked that fails, good logs, and ISO's that work fine I think I can diff things out.

What was even strange (and I don't recall if either of you had this happen), when CubicExplorer crashed it gave the the option sets window. I saved the bug file then for kicks selected the option to relaunch application and it seemed to work just fine. Did that happen for you as well or did it continue to stay crashed?

Unfortunately this weekend is a wash so I don't expect to be able to really spend the needed time working it out until next weekend or so. I won't be able to check my blog comments either until Sunday afternoon so don't worry if things go dark here for a few days.

Hang in there! I'm determined to solve this mystery for you all!

Cheers!

--Claus V.

Claus said...

@ Ryan and Anonymous - 99.99% sure I've traced down the cause of CubicExplorer working on my home tests and can explain all the failures to execute on all your varied PE builds and (finally) on a PE build I recently did as well.

Equally nice are the solutions. In fact, if I am right, you should probably be able to take any of your "failed" ISO builds and get CubicExplorer working perfectly, almost instantly. I'll be eager to hear if you also are able to meet with success.

Unfortunately for you both (and possibly more than a few lurkers as well), I still have both a shortage of time on my hands during the work-week as well as a great whodunit mystery and reveal I can't help but want to do a full blog-post on!

I want to test the solution on another home system of mine just to be sure I can replicate the issue as well as the solution and make sure it wasn't a fluke of the one I was working with (doubt it but I know a lot of folks are beating their heads against a wall with this issue). I don't want to post a solution too quickly and find out that really wasn't it.

I will say this, the CubicExplorer crash issue in PE builds has nothing to do with security rights or the amount of scratch space the PE build is configured with. It has nothing to do with the version of the PE (2.0 or 3.0, RC or not), nor the PE source files (WAIK versus OPK). Finally it has nothing to do with the version of CubicExplorer either. You all have confirmed that by your own efforts and now I understand why.

Nope. I'm 99.99% certain it is a completely different thing...and I almost had to laugh at myself once I was staring at it.

It is a real issue and has several real solutions. Plus, in the process I'll even toss in a clever "bonus" enhancement as well.

I promise it will all make perfect sense shortly as well as highlight the challenges PE builders and modders face.

Thanks for putting me on this challenge.

I hope you will be pleased with the solution when the "big reveal" comes...hopefully no later than Saturday.

Stay tuned!

--Claus V.

Anonymous said...

I'm on the verge of finding a solution for this problem, thanks to your hint, Claus.

I never thought to try saving a log dump. When I tried that, and then restarted CubicExplorer, it seemed to work fine. And it continues to work fine until the computer is rebooted and the changes to the RAM disk are discarded. Further, I noticed that even if one cancels the crash log dump (while at the Save dialog), CE still functions as expected upon reloading the application. Pretty bizarre. The fact that CE functions until a reboot indicates that something on the filesystem is changing as a result of clicking that log dump button, even when the Save operation is canceled.

I also noticed that the hash of CubicExplorer.exe changes when the log dump button is clicked. I'm not sure what to make of that.

The bottom line is that once CE is working through this "trick", it is possible to copy/paste the entire "working" WinPE contents into a new WinPE WIM, and then CubicExplorer functions as expected on every boot.

I've taken to using ProcessMonitor and ProcessExplorer (from sysinternals) in an effort to determine what changes are being made to the filesystem/registry that allow CE to work properly. I hope to make this determination sometime to day and will report back.

Maybe Claus will be me to it... =). Thanks again for all your help, Claus.

Anonymous said...

You were right, Claus. It's laughably stupid.

All I had to do was create the directory X:\Windows\System32\config\systemprofile\Desktop, within my WinPE structure.

Apparently, CubicExplorer depends on this directory's existence; if the directory does not exist, CubicExplorer will crash on launch.

Clicking the log dump button on the crash message dialog "fixes" the problem because when the "Save" dialog is launched, CubicExplorer creates the "Desktop" directory. That's why clicking "Cancel" on the save dialog has the same effect as actually saving the log. Launching the Save dialog, in which the default save location is the Desktop, is what fixes the problem.

Go in peace, CubicExplorer users, for your WinPE woes are no longer.

Claus said...

@ Anonymous - Good work!

That's it exactly.

CE expects there to be the "desktop" folder as you noted:

X:\Windows\System32\config\systemprofile\Desktop

I'm still planning a slightly more detailed blog post writeup on this but you have the root cause in hand.

I suspect that adding this folder may help additional applications work better as well under PE. Couple that with some bumps in scratch-space 256-512 MB (if system RAM is available) and you can seem to get much better performance and operations from your "portable" Windows apps under the PE environment.

Glad you feel I gave you enough clues to track it down and independently verify the solution.

Cheers!

--Claus V.