Showing posts with label command-line interface. Show all posts
Showing posts with label command-line interface. Show all posts

Friday, November 23, 2018

Win 10 Laptop Storage Space-Drain Solved!

A lifetime ago (back in July) I was struggling to understand why my Win 10 workplace laptop’s hard-drive was running out of storage space.

grand stream dreams: QuickPost: PowerShell Scripts and Win 10 Helps

Despite my best efforts the 500 GB HDD had gotten down to about 60 GB – 80 GB of free space despite some pretty severe Windows space-hog/temp-file cleaning work.

That post detailed a number of PowerShell scripts I was using to try to find out the source of the space usage.

Reminder – because this was a work-laptop, my use of the normal third-party storage analysis tools I would rely on was verboten.

After several weeks of running modified versions of many of these scripts I was nowhere closer to finding the issue. I had lots of data, but the results suggested that the file or folder where the space eating usage had happened was off-limits to typical admin-level scripting runs.

Just before giving up and planning for a system reimage, I had an epiphany; Windows 10 comes with a built-in Storage Sense platform to let you understand (at a very high-level) space usage of your local drives in a categorized manner!

How to manage Disk Space & Storage using Windows 10 Settings – The Windows Club

Go into Settings (the gear icon in Win 10), select “System”, on the left side-bar select “Storage”, then click on one of the local system drives at the top you want to explore under the “Storage” section.

Let it run for a moment and it will then give you a report of storage use.

In my case it was immediately apparent that the “System & reserved” section was where the maximum file-usage was occurring; way more than a baseline Win 10 laptop in our enterprise.

You can click on any section and dive deeper into that.

That revealed that I had over 250 GB of files related to “System Restore”.

This was an interesting find as our Group Policy turns off System Restore. A visual check of all the options and settings under “Manage system restore” confirmed these were all disabled…so what was there and how did it get there?

VSSAdmin to the rescue!

I opened an administrator-elevated CMD window and ran the following command:

vssadmin list shadows

That revealed one “orphaned” shadow copy file created quite some time before I actually noticed the space drain.  A check of our operations logs for that timeframe couldn’t find any obvious actions or infrastructure routines that should have created it. So how it was triggered remains a mystery.

With the shadow copy ID known now I next ran this, using the actual ID # I found for that argument variable:

vssadmin delete shadows /Shadow={shadow copy ID}

It ran for quite a while, but then returned back to the blinking prompt cursor.

Another “vssadmin list shadows” confirmed it was gone, and a look at the local drive properties showed my space usage was back to a normal level with lots of free space available again.

Mischief managed!

More resources:

Cheers!

Claus V.

Sunday, July 29, 2018

QuickPost: PowerShell Scripts and Win 10 Helps

Since having a system migrated to Windows 10, I’ve noticed a trend of the hard-drive getting significantly fuller now.  I’ve done all the standard post-migration cleanups. I cleared over 40 GB of old software packages off the system and was feeling pretty good.  The next day all my space-gains were lost and I was back at 100 GB of free space where I started.

I suspect there is some caching activity going on in the background and that it running off a quota that keeps me returning to the magical 100 GB free of a 500 GB drive.

Normally, I’d just run one of these tools to identify the space/file hogs and start cleaning up. I’ve ordered these in my general preference; though I like them all for slightly different things they bring to the table on a space-hunt.

However in this case I cannot use any third-party tools and must stick with using Microsoft OS-based solutions only.

So that led me to find a script I could use in PowerShell.

I’ve divided them into file-size analyzers and folder-size analyzers.

I found it is relatively easy to hunt down singular files on your system in PowerShell that are the largest. However, what happens if you have a bunch-load of very small files? Individually they may never float to the top, however in aggregate, they could add up to a lot of space usage.

I’ve listed these as well in my order of preference.

Note: They all seemed to run fine on my Win 10 systems in PowerShell ISE – though tweaking was needed for each one to target specific folders and/or report outputs – depending on the script.

PowerShell File-Hog Hunters

PowerShell Folder-Hog Hunters

If you do export output to CSV and don’t “pre-format” the bytes output to MB, here is a tip on a custom formatting rule in Excel you can use to make it more readable.

formatting - How can I format bytes a cell in Excel as KB, MB, GB etc? - Stack Overflow

I’ve not loaded Ubuntu on Windows to have a Bash console, but in looking for tools, I came across this that looked pretty neat: ncdu: Identify Large Files on Windows 10 - Trevor Sullivan

Finally, on Win 7 I used a pretty small set of common keyboard shortcut to navigate my way around the system.  In Windows 10, I’m finding a desire to expand my quick-access key combo skills. Here are some good resources:

If I’ve missed a useful script or you have any tips for hunting for space/file hogs using only “on-board” native Windows 10 OS tools, please drop a comment!

I expect I’ll be adding to the list of links in this post too as I uncover more PowerShell scripts that could be useful. As I post this, I think I am overlooking one or two others that I found useful

Cheers!

--Claus V.

Monday, May 30, 2016

Sysadmin Tools and Tips Linkfest: Part II

Mores…

Windows Updating and Patching – Tips and Tricks

Windows Troubleshooting and Tips

Windows Tools

Windows PowerShell

Microsoft News Bits

Cheers,

Claus Valca

Quickpost: that whole Bash on Windows thing

Old news; just a collection of links for future reference.

Moving on…

Claus Valca

Saturday, February 13, 2016

PowerShell one liner to get USB Flash Drive Serial Number

First a confession, I bought the “Learn Windows PowerShell 3 in a Month of Lunches” book quite a while ago but still haven’t had the discipline to work my way through it.

Couple that with tons of PowerShell learning resources documented across the GSD blog and I am woefully behind the game.

However, I can read music so I guess that means I can puzzle out code with patience and Google searching.

One of my duties is to keep inventory of USB device issuances to the team. I record the serial numbers of the devices in a spreadsheet.

While some external USB HDD’s have the device serial number listed on the external case, USB flash drives aren’t so good wtih that practice.

There are some USB imaging tools such as the USB Image Tool or utilities such as USBDeview or Kanguru’s free Serial Number Display Tool that I could use to get that information directly.

However, I really like to stick with “in-box” tools and techniques so while that would be the “easy” way I’ve been collecting the information the “long-way-round” by the following process:

  1. Insert USB flash drive into system.
  2. Open Control Panel
  3. Open Device Manager
  4. Expand the “Universal Serial Bus controllers” item
  5. Look for the “USB Mass Storage Device” in the list (that’s my inserted flash drive)
  6. Right-click that item and select “Properties”
  7. Select the “Details” tab
  8. In the Properties list, pick “Device Instance Path”
  9. Read the serial number which is the trailing string after the last “\”

Eject and repeat.

What I wanted to do was just run a “WMIC” query directly from the command line to get that specific information via the Command Prompt.

But despite my best efforts (which isn’t saying much) I wasn’t able to locate the correct parameter.

Tonight I did find and tweak a similar command via PowerShell and it does the trick. So while it isn’t elegant, it does the job I need. Victory at last!

gwmi Win32_USBControllerDevice |%{[wmi]($_.Dependent)} | Where-Object {($_.Description -like '*mass*')} | Sort Description,DeviceID | ft Description,DeviceID –auto

(That’s one long string btw.)

So I can get a result like this:

Description             DeviceID
-----------             --------
USB Mass Storage Device USB\VID_0DD8&PID_3200\<serial number displayed here>

Your DeviceID details will vary depending on the make/model/etc. but the serial number is there easy to copy/paste.

Credits:

wmi - Getting a USB Device Instance Path in Powershell - Stack Overflow. This post got me finally on the right track but still didn’t work for me.

Displaying USB Devices using WMI - Windows PowerShell Blog.  Jeffrey Snover’s post did get me the output I was looking for, but I then needed to clean it up just a bit as I only wanted the “USB Mass Storage Device” information.

Filtering PowerShell Objects - PowerShell content from Windows IT Pro. This was the final bit in giving examples how I could filter for just the one line I needed.

If anyone can “refine” the code or filter it more to just pass on the Serial Number string itself, I would be grateful for any recommendations.

Using PowerShell’s gwmi or the Command WMIC call are powerful ways to get system information.

Here are some additional examples that I’ve collected trying to puzzle out my own particular USB Mass Storage device query technique that could be helpful to sysadmins:

Cheers!

--Claus Valca.

Bonus USB utility finds:

Windows Update Agent Utility (WUAUclt.exe)

Just some sysadmin notes…nothing to see here.

  • Windows Update Agent Utility (wuauclt.exe) - pauby.com
  • wuauclt - Windows Update - Windows CMD | SS64.com
  • Appendix H: The wuauclt Utility – Windows Server TechNet page
  • wuauclt commands to force reporting of status? – Windows Server TechNet page
    • The /reportnow function is a very tricky beast, and it somewhat requires an understanding of the natural behaviors of the WUAgent.

      When the WUAgent performs activities, it queues up all of those completed activities as 'events'. When the WUAgent quits working, an idle timer is engaged, and when the WUAgent has been idle for ~20 minutes, it invokes a call to the ReportingWebService. You can see these calls in the WindowsUpdate.log and compare their timestamps with the entries just previous.

      If the /reportnow action is invoked after the WUAgent becomes idle and before the regular call to the ReportingWebService, an immediate call to the ReportingWebService will be invoked. You can also see this in the WindowsUpdate.log.

      However, the call to the ReportingWebService is not the end of the line. Those events reported in that call are loaded into a buffer, and the WSUS server then processes those events asynchronously. If the WSUS server is also busy doing other things or other clients are also reporting at the same time, there will be an additional delay until the results are visible in the WSUS console.

      The real thing to remember is that, at best, the /reportnow task isn't going to save you much more than 20-30 minutes, so usually just waiting is the more efficient approach to the whole thing.

      as commented by Lawrence Garvin

  • wuauclt and other questions about WSUS – Windows Server TechNet page
  • When run from the command prompt on a client in the form of "wuauclt.exe /resetauthorization /detectnow", this command will kick off a manual check in with the configured WSUS server (or WindowsUpdate website, if you're using that instead of WSUS).  You can verify this occurs by opening the windowsupdate.log file located in the Windows directory.

    If you manually run wuauclt.exe with the above listed switches, it will check in with WSUS, and then behave based on your update configuration, either configured locally on each client, or through Group Policy.  So, if you have your clients configured to download any available updates, it will do so.  Or, if you have your clients configured to just check in and inform you there are updates without downloading, it will do that also.

    The windowsupdate.log file, located on the individual clients, will have all the information pertaining to how wuauclt runs on their individual systems.

    as commented by John

    and…

    To expand on John's answer...

    There are three functional uses for the WUAUCLT.EXE command.

    1. wuauclt /detectnow  -- initiates a detection event to the assigned update services resource (AU or WSUS).

    2. wuauclt /resetauthorization /detectnow -- this is actually a special case version of the previous command. The /resetauthorization parameter forces the targeting cookie to be immediately expired. Normally the cookie has an ~60 minute expiration. Typically this form of the command is used when server-side targeting is being used, and a client system has just been reassigned to new group(s) via the WSUS console. Use of this command forces the WUAgent to discard any previously known group memberships and to requery the WSUS server for the current memberships. This command should also be used when the SusClientID has been deleted and a detection was performed within the previous hour to ensure the WUAgent does not use the SusClientID that is cached in the targeting cookie. Also note that the order of these parameters is critical -- the /resetauthorization flag must be the first of these flags on the command line.

    3. wuauclt /reportnow -- IF a recent event has completed and there are PENDING EVENTS to be reported to the WSUS server, this command will force the immediate flushing of that reporting event queue. If there are no pending events to be reported, this command does nothing.

    No other parameters are supported or documented -- although many are defined in the source code and have been extracted via reflection. Some of them have experimental functionality, but should not be used in a production environment.

    as commented by Lawrence Garvin

And the commands for copy-paste.

  • wuauclt /ResetAuthorization /DetectNow
    • wuauclt /a /d
  • wuauclt /ReportNow
    • wuauclt /r

moving on…

--Claus Valca

Saturday, November 28, 2015

Windows Updating Fixes - Maybe

I have two Windows 7 Pro x64 systems up at the church-house that refuse to comply and install IE 11.

I keep reviewing the install log at C:\Windows\IE11_main.log for failure analysis and all the requirements seem to be present, but it still fails; both via Windows Updates or a manual (re) installation of all the pre-requisites and the main IE 11 install file.

(When I find where I put those log file captures I’ll update the post with more detail.)

Update: Found my log file capture! Install error 09C57

This is from just one of the two machines I’m having the same issue with. I need to pull the second system’s IE install log and compare to see if they match. I suspect they will.

It doesn’t matter if I run either the manual IE 11 installer package or the Windows Update obtained package…results end the same.

I’m cleaning up the log file and picking only some of the key lines from it for brevity.

  • Command line: "C:\Users\profile\Downloads\IE11-Windows6.1-x64-en-us.exe"
  • iexplore.exe version check success. Install can proceed.
  • Updated Download list, Hardware Blocking list, and no reboot policy files successfully downloaded and extracted for use.
  • Launched program to check hardware: "C:\Windows\TEMP\IE1924.tmp\IE11-SUPPORT\IEXPLORE.EXE" /CheckHardware "C:\Windows\TEMP\IE1924.tmp\IE11-support\HardwareBlockingList.xml"
  • Graphics Device Information: NVIDIA Quadro NVS 295
  • Hardware support check succeeded. Installation will continue.
    <cv note: all 9 of the packages download fine>
  • Prerequisite download processes have completed. Starting Installation of 9 prerequisites.
  • Launched package installation: C:\Windows\SysNative\dism.exe /online /add-package /packagepath:C:\Windows\TEMP\IE1924.tmp\KB2834140\Windows6.1-KB2834140-v2-x64.cab /quiet /norestart
  • Process 'C:\Windows\SysNative\dism.exe /online /add-package /packagepath:C:\Windows\TEMP\IE1924.tmp\KB2834140\Windows6.1-KB2834140-v2-x64.cab /quiet /norestart' exited with exit code 0x800F081E (-2146498530)
  • Error installing prerequisite file (C:\Windows\TEMP\IE1924.tmp\KB2834140_amd64.MSU): 0x800f081e (2148468766)
  • PauseOrResumeAUThread: Successfully resumed Automatic Updates.
  • Setup exit code: 0x00009C57 (40023) - Prerequisites failed to install.

I had tried to manually install each of the prerequisite files and don’t remember having any issues though I seem to recall when getting to KB2834140 it said it wasn’t required/needed on the system and exited.

The key clue is “exit code 0x800F081E (-2146498530)” which I understand means "The specified package is not applicable to this image".

Maybe I’ve got the system looking at and pulling a corrupted WSUS update?

Anyway…I’ll be coming back to this with a fresh post soon. Chasing down possibilities from these error codes quickly becomes an Alice in Wonderland adventure.

Likewise, at work found that some of our Surface Pro 3 tablets just don’t want to pull updates down from the WSUS.

Normally when a system is “stuck” getting updated WSUS patch delivery we might escalate getting it going (after confirming it is correctly pointing in the Registry) by first doing a “gpupdate /force”. I know.

If that doesn’t work we next try the following.

  1. wuauclt /resetauthorization /detectnow
  2. (wait 5-10 minutes)
  3. wuauclt /reportnow

If failing that, this routine comes.

  1. Stop the Automatic Updates service
  2. Rename the software distribution folder (i.e. C:\Windows\SoftwareDistribution).
  3. Restart the Automatic Update service
  4. wuauclt /resetauthorization /detectnow
  5. (wait 5-10 minutes)
  6. wuauclt /reportnow

Only that still didn’t work on the Surface Pro 3’s.

I really hoped not to reimage the systems as a “troubleshootin” solution as that is a lot of work and user impact so…I managed to get them working thusly.

I had checked the C:\Windows\WindowsUpdate.log and found a particular error that came up after each “manual” update refresh attempt.

That led me to this solution.

Since it was a Windows 8.1 system I ran the following command.

DISM.exe /Online /Cleanup-image /Restorehealth

It found an error with an AMD-related update package component (go figure) and repaired it.

After it completed, I did a system reboot and the updates have flowed faithfully since.

If you have a Windows 7 system, then you can use this System Update Readiness tool.

More Information and additional tools and tips:

None of those have helped with my IE 11 installation issue but with IE 8 (that it is stuck on since IE 9-10 also won’t go on) retiring soon, I’m determined to get it fixed once and for all.

Cheers!

Claus Valca

Gpupdate /mostlyfineasis

One of the cure-all’s we use at work for fixing issues with our AD-joined systems is to run the following command:

gpupdate /force

Fixes everything! (not)

So when this Helge Klien post came in this week with it’s incendiary lead-in, I had to pay attention.

How Group Policy Impacts Logon Performance #1: CSEs • Helge Klein

Gpupdate /force is for wimps!

Say you have changed a Group Policy setting in the domain and want to test its effects on a member computer. You open a command prompt and type:

gpupdate /force

Please pause and think this over before hitting enter. Why the /force switch? To show that stupid machine who is its master? Are you one of those people that click Apply before they click OK? Do you wear both belt and suspenders? Of course you do not! So let us take a look at the help text for the /force parameter:

Reapplies all policy settings. By default, only policy settings that have changed are applied.

That is quite telling. Group Policy keeps track of what has been applied and does not reapply settings that are already present. Nice! So why would we override this optimization? We would not. Using /force typically is only required when your Group Policy infrastructure (i.e. AD and/or DNS) are broken. Go fix it instead of telling poor old Group Policy to forego optimizations!


That got me digging just a little bit deeper into the whole gpupdate /force thing we do and tell others to do without consideration.

GPUpdate or GPUpdate /force? Learn the difference! - DeployHappiness.

From Joseph Moody’s entertaining post:


As it turned out, Group Policy was always working – I just didn’t understand it. So what’s the difference between GPUpdate and GPUpdate /force? Well –

GPUpdate: Applies any policies that is new or modified

GPUpdate /force: Reapplies every policy, new and old.

So which one should I use? 99% of the time, you should only run gpupdate. If you just edited a GPO and want to see results immediately, running gpupdate will do the trick. In fact, running GPUPdate /force on a large number of computers can be damaging to your career. This is because these machines will hit a domain controller and reevaluate every GPO applicable to them.

Note: If you are looking for the remote version of GPUpdate in Active Directory Users and Computers, see this guide.

Anything else?

Since you asked, why yes there is! GPUpdate has a few other options for you to use.

/LogOff: Certain GPOS, such as Folder Redirection, can’t apply in the background. If a logoff is required, this switch will initiate it.

/Boot: If a policy, such as software installation, needs to be applied – the boot command will reboot the machine.

/Sync: Useful for changing the foreground (startup/logon) processing to synchronous.



Also handy after running a “gpupdate” command:  “gpresult /h c:\temp\myGPreport.html”


Open with IE or Chrome or something and check your results of applied policies. If you use this technique, you can quickly search through the results with the search feature of the browser to highlight and jump to specific policy references you are looking for. It is a real time saver



Bonus Material



Cheers,


Claus Valca.

Tuesday, July 28, 2015

Sysadmin Link Seventh-Inning Stretch

Here are some tips and tricks for the sysadmin crowd.

While doing a project on a Win 7 laptop cleanup, I was looking for an automated way to clean off all the inactive/unused Windows user profiles. Sure I could have gone into the advanced settings and removed the account profiles manually. But a command-line tool would have been helpful.

I re-found the Delprof2 - User Profile Deletion Tool provided by Helge Klein. It worked as advertised. The first pass I used it it could not delete one local profile for some reason. Turns out that some software that had been installed was still running as service under that account. After I had deleted the software and rebooted, the tool then worked to remove that remaining account. Free for private, non-profit org, or educational org use. Otherwise requires a commercial license purchase.

More Tips and Tricks

Smartphone-Friendly Conference Bridge URL Formatting

One of my biggest frustrations with conference call invites is receiving one when I am in the field and having to join via smart-phone. The meeting reminder comes up with the main dial-in number, but then I have to try to switch back-and-forth to find, note, and enter the actual bridge #.

So I found a standard formatting that can be included/used for one-click use in both calling the main # and then auto-entering the bridge number.  Please folks! start adding this to your meeting invites!

The basic format is thus:

tel:12345678,,100200#
join 100200 conference code on the conference line 12345678 on most of the newer devices

Malware Anti-Exploit Update

Malwarebytes Anti-Exploit - Version 1.07.1.1015 was released. From the setup installer’s change notes:

Malwarebytes Anti-Exploit 1.07.1.1015

New Features:
• Added new Layer0 exploit mitigations for IE VB scripting
• Added new Layer1 exploit mitigations for ROP detection
• Added new Layer3 exploit mitigations for Powershell abuse
• Added telemetry from Firefox
• Added ability to edit custom shields
• Added ability to log protection events to UI
• Added ability to auto-upgrade corporate builds
• Added support for Windows 10
• Added blacklisting of pirated and fraudulent license keys

Improvements:
• Improved Java shield in corporate environments
• Improved exploit telemetry
• Removed duplicate default shields for portable browsers
• Removed "shielded applications" counter from UI

Fixes:
• Fixed issue when printing to Adobe PDF
• Fixed issue with Speedbit Download Accelerator
• Fixed issue with plugins from PowerDVD and GAS Tecnologia
• Fixed issue with nProtect GameGuard Anti-Cheat
• Fixed issue with certain exclusions not respected
• Fixed issue with Knowledge Coach Office Add-In
• Fixed issue with false positive from IE
• Fixed issue with Foxit Reader startup
• Fixed issue with Excel PowerQuery
• Fixed issue with Excel DEP Enforcement
• Fixed issue with IE VB scripting block
• Fixed issue with Chrome crashes

Techniques for adding “Open Command Prompt Here” &/or “Elevated” to the Windows Explorer Shell Menu

Windows 8/8.1/10 have an option to allow you to open both a Command Prompt or Elevated Command Prompt window from the start menu.

You can also right-click a folder or white-space and open a command window. However you likely won’t be able to open an elevated one easily.

There are a number of ways you can modify the registry to create some optional Explorer shell menu items. And you can do some clever keyboard/copy/paste tricks as well in the default Windows GUI.

In the end I found and went with this utility on my personal systems.

Here are a few more options:

On my own system I don’t mind using a utility to make the changes needed, but if you really need the feature and are authorized to make the changes, “manually” setting the feature via RegEdit works well. The benefit of that method is that you “know” what changes are being made and how to remove/regress them if required.

Windows Critical Out of Band Security Patch Released

Yes…late again…but better late than…well, you know.

Cheers!

Claus Valca

Sunday, May 03, 2015

Thoughts on USB drive duplication for Sysadmin work

Deployment of system images has matured over the years at the sausage factory.

In the wild-west days, we would have just a handful of hardware platforms. So when they came off the truck from the factory, we would use Clonezilla to grab an image of the system.  As time went on we would periodically apply the respective image back to each of the platforms, run all our update processes, do a new Sysprep shut it down, capture an updated image, rinse, repeat.

I later shifted image capture from Clonezilla to Microsoft’s ImageX platform and could capture multiple platforms in a single WIM and deploy it depending on the specific hardware. That really helped cut down on having “separate” image DVD’s and external HDD’s to carry around and saved a lot of space due to the way ImageX and WIM files worked.

As the organization grew (became more departmentalized and specialized) other teams gradually took over the imaging process and standardized it.

Today we no longer handle the image building and distribution work ourselves but get the latest images built and shipped to us on a master USB stick.

Hopefully we will move on from this method to PXE booting and image deployments. Won’t that be exiting!

So now the challenge has shifted from system image duplication and distribution to image device duplication and distribution.

Typically the method involves taking the (bootable) USB master stick, and capturing a single IMG file of the entire device, then writing that disk image back to all the target drives.

I’ve struggled with this method for a number of reasons:

  • As the USB sticks have gotten larger, the time to capture an image of the drive has gotten longer.
  • As the USB sticks have gotten larger, the time to write an image of the drive to other devices has gotten longer.
  • It takes a lot of space to store these IMG files; particularly as I tend to hold on to a few of the previous versions--just in case.
  • Since we rarely use the entire space on the USB drive, there is a lot of wasted space and time capturing and then re-writing back that unused space.
  • As different USB sticks are procured, variances in actual space on disk can wreak havoc when your master stick turns out to be a “larger” 64 GB USB stick than the purchased 64 GB USB stick you were given.

The popular tool we were asked to used for this duty was PassMark Software’s ImageUSB.  This tool is “standalone” software, works great on capturing a single IMG file of a removable USB device, and handles writing the image back great; even to multiple devices. So if you set up a bunch of USB hubs you can quickly write the image back to them all. Well, assuming that the IMG file represents a “smaller” USB storage amount than the devices you are writing back to.

I also like Alexander Beug’s USB Image Tool which can capture both device and volume-based images of removable USB devices. It has a few more bells and whistles.

Steve Si also details how you can use RMPrepUSB to capture a more efficient image of a USB drive in his Bulk duplication of USB drive images post.

In my post All Kinds of USB Cloning Challenges… I outlined how I had used both ImageX to capture a WIM-based image of the master drive and then after some DISKPART and BOOTSECT work, restored the WIM image to the drives for cloning. That works as well and saves a ton of time and space overhead storing the files that make up the master image stick rather than taking\writing a sector-based device image of the master.

I really prefer to stick with Microsoft based solutions where I can due to the health-department standards of system cleanliness on the sausage factory floor. There is always a risk bringing in and using third-party tools and I’m not a coder so I can’t audit the code these apps run on. Can’t be too safe I suppose. So I continue to fiddle around with alternatives to the WIM based image capture/cloning method.

I’ve got a library of WIM images of these master USB images and they are pretty small. They are stored on a external USB drive. However sometimes I don’t want to go through the extra steps of connecting up the drive, authenticating past it’s encryption, and then deploying the WIM to the USB stick after re DISKPART prepping the target device.

It takes me about a minute tops to go through the IMAGEX steps to re-prep a USB drive I want to apply an image to:

And did this from the command line window.

>DISKPART

DISKPART>LIST DISK

note: this is to confirm which disk number the USB drive was showing at…in this case the USB stick was disk 1 as my system disk was disk 0. Be sure you get this part identified correctly or bad things can happen! Your system will almost certainly vary!

DISKPART>SELECT DISK 1

DISKPART>CLEAN

DISKPART>CREATE PARTITION PRIMARY

DISKPART>ACTIVE

DISKPART>ASSIGN LETTER = E

DISKPART>EXIT

>format E: /fs:ntfs /q /y

note: at this point I have a freshly formatted (smaller) USB stick that is empty. Next I need to make it “bootable” so the USB stick will work as designed for system booting/imaging after the files are restored. I used the bootsect.exe tool to do this. You should be able to find it under the Windows ADK that you probably installed to get to this point to first have snagged imagex.

> bootsect /NT60 E:

Then I have just used a working “master” USB stick I had cloned off the true USB master stick I got and do a simple XCOPY to copy the files from the master to the copy.

XCOPY D:\*.* E: /e

That seems to work pretty well and duplicate USB sticks perform as needed.

I’ve also seen a variant command thusly:

XCOPY D:\*.* E: /e /h /k

The “/e” switch makes sure that directories and subdirectories -- including empty ones -- get copied.

The “/h” switch makes sure that hidden and system files get copied.

the “/k” switch makes sure that attributes get copied also as the normal xcopy will reset read-only attributes.

More here: MS-DOS xcopy command help - Computer Hope

However, Win 7 and up also now come with RoboCopy as part of the arsenal.

So I should be able to do the DISKPART parts, then use the following command to make my clone:

At the simplest I could do the following:

Robocopy D:\ E:\ /E

But since I am really trying to make a mirrored duplicate of the “master” I could do thus: 

Robocopy D:\ E:\ /MIR

At this point it seems that the biggest choke-point in the image clone deployment process is the speed at which the system can shuffle data through the USB port(s) and onto the USB stick itself. Man those things get warm when done!

And here is the other thing.

Our new Windows 8 Surface Pro 3 tablets use a different USB stick for their image/deployment.

And they aren’t just any “removable” USB stick. Nope. They have to be seen as “fixed disk” type USB drive hardware as they contain two separate and distinct partitions. Windows can’t recognize those on “removable” type USB flash drives.

So out goes PassMark’s ImageUSB tool as it can’t handle the fixed disk type USB devices with more than one partition. Nor does USB Image tool seem to handle device duplication of these multi-partition fixed disk USB drives. At least as far as I have been able to figure out.

However, being able to see and replicate the fixed-disk USB flash drive partition structure and make it bootable using just DISKPART and BOOTSEC, AND being able to restore the files to each of the partitions using either IMAGEX WIM deployments, XCOPY, or ROBOCOPY these methods make duplication of even these “fancy-pants” USB image deployment master sticks pretty straightforward.

Now, if I can only convince the powers that be to invest in USB 3.0 drives and hardware platforms that use USB 3.0 ports, I should be able to really cut down the image capture and image/file writing times for duplication.

If you have any other suggestions or recommended additional switches for use in these scenarios with XCOPY or ROBOCOPY please drop some tips in the comments.

I’d love to refine the process and steps even more.

Cheers!

Claus Valca

Sunday, August 10, 2014

I’m sure there is a better way to accomplish this…

In my GSD blog post Anti-Malware Response "Go Kit" I outlined a variety of tool-sets and standalone tools that I carry on my USB flash drive for dealing with malware responses on friends/family systems.

Keeping the IR tool-sets (Confessor, MIR-ROR, rapier, TR3 Tool Kit v2, and triage-ir) updated is a lower priority for a number of reasons.

  1. It’s a lot of work,
  2. the developers often require (due to licensing) the end user (me and you) to download the supporting binaries directly from the developers’ sites, and
  3. you always run the risk that a later utility update may break the way the scripts run on the package.

None of those are deal-breakers, but because of that, keeping those updated (aside from the main IR package) leads me to not update them as frequently, maybe once a quarter to biannually.

The ones that I do update frequently are the ones that are used to to sweeps for malware and/or viruses.

Most of these are signature based, and if they are updated, there there is a high likelihood a scan with an older tool may miss something critical!

So to keep them updated, I have a bookmark folder with URL links to all the tools. I then go down the list, click, download, copy to USB, rinse and repeat.

So yesterday I wondered if I could automate the process a bit. Kind of like a poor-man’s version of NirLauncher or KLS SOFT’s WSCC - Windows System Control Center.

I’m sure there is a better way to do this, but this was my “it works for me” result.  I’m not posting the actual files (at least in fullness for now) but will show you the basics so you can build your own if you want.

First, I considered (and may still go to) a process/script that uses Wget for Windows - GnuWin32.

But I wanted to start with what I knew (or thought I did) for now.

To get the ball rolling, I made a “landing zone” folder on my Windows system at C:\TEMP\AMW_Packages

This is where I wanted to download the updated files into. I wanted to keep it separate in case I decided I didn’t want to end up overwriting any of my previous files. So once all the packages are downloaded here, I will manually copy them over onto my USB drive folder where they reside full-time.

I then created a Windows BAT file called “a-AMW_Package downloader.bat”

It does a few things.

It deletes all the files/folders in the “C:\TEMP\AMW_Packages” location to get a clean start.

It then runs down a list of the utilities I need to get/update, and downloads them into the “C:\TEMP\AMW_Packages” folder using PowerShell. (I know! Cool!)

Then, there are some packages that have some fancy dynamic page tricks/EULA’s that make getting those binary files a bit of a hassle. Some of those I was able to work around with the PowerShell commands below. However others were not so cooperative. And that was OK.

So at the end of the BAT file, it calls a custom EXE called “a-BAT-IECall.exe”.  That file was a different PowerShell script block I came up with to open up all those “problem” site URL’s in a single Internet Explorer window session, each in a different tab; more on it in a bit.

The resulting automatically opened IE window allows me to review/download those “manually” as needed.  (I guess I could put it at the front so I could be manually downloading those as the script continues to run in the background. But this made sense to me. I also dropped some FYI URL pages as well there to remind me of some tricks I keep forgetting or to see if any new tools are available that I may want to add to my tool-kit.

Here is an abbreviated version of the BAT file “a-AMW_Package downloader.bat” contents. You should be able to get the gist of what I am doing and add more lines for other resources you may want/need.

:: Anti-Malware Response “Go-Kit” Downloader

:: Clean Up Download folder first

set folder="C:\Temp\AMW_Packages"
cd /d %folder%
for /F "delims=" %%i in ('dir /b') do (rmdir "%%i" /s/q || del "%%i" /s/q)

:: Now Let's Get the Files!

:: Process Explorer
powershell -Command "(New-Object Net.WebClient).DownloadFile('
http://download.sysinternals.com/files/ProcessExplorer.zip', 'ProcessExplorer.zip')"
powershell -Command "Invoke-WebRequest
http://download.sysinternals.com/files/ProcessExplorer.zip -OutFile C:\temp\AMW_Packages\ProcessExplorer.zip"

:: AutoRuns
powershell -Command "(New-Object Net.WebClient).DownloadFile('
http://download.sysinternals.com/files/Autoruns.zip', 'Autoruns.zip')"
powershell -Command "Invoke-WebRequest
http://download.sysinternals.com/files/Autoruns.zip -OutFile C:\temp\AMW_Packages\Autoruns.zip"

:: Microsoft Safety Scanner & Malicious Software Removal Tool  (+ download others manually in a bit)
powershell -Command "(New-Object Net.WebClient).DownloadFile('
http://go.microsoft.com/fwlink/?LinkId=212732', 'msert.exe')"
powershell -Command "Invoke-WebRequest
http://go.microsoft.com/fwlink/?LinkId=212732 -OutFile C:\temp\AMW_Packages\msert.exe"

::And so on, and so forth for all the other tools as needed

:: Trend Micro Anti-Threat Toolkit  (download manually in a bit)
:: VIPRE Rescue (download manually in a bit)
:: AdwCleaner (download manually in a bit)
:: ComboFix (download manually in a bit)
:: Rootkit Buster - x86 - Trend Micro  (download manually in a bit)
:: System Explorer (download manually in a bit)

:: Misc Tools and Utilities (Now we fire up IE via a PowerShell script) so we can launch IE and the link URLs in tabs for manual download if we need them.

a-BAT-IECall.exe

Exit

Just add more of those download lines for all the tools you need as long as the URL download links are functional with this method.

So next, about that “a-BAT-IECall.exe”

This took a bit of creative work to generate.  There are other ways to launch IE in a standard BAT file, but it ended up opening each URL in a separate IE window that cluttered up my system, despite my best attempts. So this way worked perfectly, and because: PowerShell!

The PowerShell script that is the heart of the engine looks like this:

$ie = New-Object -ComObject InternetExplorer.Application
$ie.Navigate2("
http://systemexplorer.net/download.php")
$ie.Navigate2("http://www.vipreantivirus.com/live/",0x1000)
$ie.Navigate2("https://toolslib.net/downloads/viewdownload/1-adwcleaner/",0x1000)
$ie.Navigate2("http://www.bleepingcomputer.com/download/combofix/",0x1000)
$ie.Navigate2("http://www.bleepingcomputer.com/forums/t/403413/cannot-execute-exe-reg-regedit/",0x1000)
$ie.Navigate2("http://free.antivirus.com/us/rootkit-buster/index.html",0x1000)
$ie.Navigate2("http://www.microsoft.com/security/scanner/en-us/default.aspx",0x1000)
$ie.Navigate2("http://www.microsoft.com/en-us/download/malicious-software-removal-tool-details.aspx",0x1000)
$ie.Navigate2("http://windows.microsoft.com/en-us/windows/what-is-windows-defender-offline",0x1000)
$ie.Navigate2("http://esupport.trendmicro.com/solution/en-us/1059509.aspx",0x1000)
$ie.Navigate2("http://support.kaspersky.com/viruses/utility",0x1000)
$ie.Navigate2("http://firesage.com/mbrwizard.php?x=4x",0x1000)
$ie.Navigate2("http://freeofvirus.blogspot.com/2009/05/remove-fake-antivirus-10.html",0x1000)
$ie.Navigate2("http://www.bleepingcomputer.com/download/rkill/",0x1000)
$ie.Navigate2("http://www.bleepingcomputer.com/download/unhide/",0x1000)
$ie.Navigate2("http://www.bleepingcomputer.com/download/windows/security-utilities/",0x1000)
$ie.Navigate2("http://support.microsoft.com/kb/299357",0x1000)
$ie.Visible = $true
stop-process $PID
#

Add/remove/change URLs accordingly.

To create the EXE version for this PowerShell script to include the call to in your main BAT file:

  1. Edit the PowerShell script block above to add/change/remove any URLs
  2. Save it somewhere for quick future re-editing.
  3. Launch the PowerGUI Script Editor.
  4. Create a new workspace tab.
  5. Copy those lines into it.
  6. From the menu bar select “Tools” then “Compile Script…”
  7. Select where you want to save it...should be same place as the “a-AMW_Package downloader.bat” BAT file…and what name you want to give it, for me I used “a-BAT-IECall.exe”
  8. I left the Target framework set at “Microsoft .NET Framework 4.0” for my system.
  9. I guess you could give it a cool custom icon if you wanted. I didn’t for now.
  10. Select “OK” and let it build!
  11. Close stuff out when done.
  12. Find/test! (see result below)

3kq2ba2q.oox

Now, when I want to update my IR package tools, I just fire off the main BAT file and away it rips, leaving me to manually download just a few packages myself -- if desired -- from IE.

Misc Notes and references.

I wasn’t aware until composing this post that the PowerGUI project appears to have been taken over by Dell: Welcome to the New Home of PowerGUI. OK.

I’m sure there are WAY BETTER ways to deal with this with Wget, PowerShell, BAT files, but this works and I learned a lot in the process. I’m open to recommendations/suggestions.

The PowerShell commands in the BAT file are pretty flexible.

powershell -Command "(New-Object Net.WebClient).DownloadFile('URL-path-for-binary.file', 'binary.file’)"
powershell -Command "Invoke-WebRequest
URL-path-for-binary-file -OutFile C:\temp\AMW_Packages\binary.file"

This one is straight forward with the pattern:

:: AutoRuns
powershell -Command "(New-Object Net.WebClient).DownloadFile('
http://download.sysinternals.com/files/Autoruns.zip', 'Autoruns.zip')"
powershell -Command "Invoke-WebRequest
http://download.sysinternals.com/files/Autoruns.zip -OutFile C:\temp\AMW_Packages\Autoruns.zip"

I was able to make changes to some of the “binary.file” names to change the name as it got saved, and in some cases the URL path didn’t actually contain a binary.file name in the path but it still handled it OK. Once you have the format down you can experiment a bit. See below for one example:

:: Microsoft Safety Scanner & Malicious Software Removal Tool  (+ download others manually in a bit)
powershell -Command "(New-Object Net.WebClient).DownloadFile('
http://go.microsoft.com/fwlink/?LinkId=212732', 'msert.exe')"
powershell -Command "Invoke-WebRequest
http://go.microsoft.com/fwlink/?LinkId=212732 -OutFile C:\temp\AMW_Packages\msert.exe"

Here are the URL’s of many of the sites/tips I reviewed to get me to this stage, and a few that I wanted to do but couldn’t quite get to work like I wished.

Again, any tips, tricks or alternative suggestions would be appreciated!

Cheers!

--Claus V.

P.S. Microsoft has a number of tools for scanning/removing malware from a system.

Microsoft Malicious Software Removal Tool - This is on most all Windows systems as the MRT.EXE file. Type “MRT.exe” in the RUN bar and you will be off to the races (assuming Windows Updates are current, otherwise download the file manually above or effectiveness will be diminished.)

Then there is the heavy-duty version Microsoft Safety Scanner which gets updated every 10 days.

And, in my original post I mentioned the Microsoft Standalone System Sweeper from Microsoft that was available only via the Microsoft Connect site unless you went through a third-part download source. See this Utility Spotlight: Repair Your PC Infection from TechNet Magazine to get more info on it.

Working on the URL/Download location for this one led me to discover the Windows Defender Offline tool that may have replaced (?) the Microsoft Standalone System Sweeper.  This one is for most “modern” versions of Windows but if you are running Windows 8.1 you will need to jump to this Windows Defender Offline Beta build page.

--cv

Saturday, December 01, 2012

Brief list of port monitoring tools

A while back we deployed some special purpose server systems. They are pretty simple in terms of function but do provide some critical infrastructure support.

So when they go off-line for whatever reason (power failure, unplugged network cable, etc.) we need to respond to get them back online.

Proactive monitoring is pretty thin and currently we have more of a reactive monitoring solution. Someone needs one of these systems, finds it is down, and call us to fix it fast.

Nice.

So one solution developed was as simple batch file that ran a ping against the server IP’s. If a ping failed, then notice would be auto-emailed to selected staff to check it out.

That seems ok, but what happens when the NIC is up and responding to ping, but the core applications/OS has actually hung and it really isn’t “operationally” on-line, though the NIC is? Kinda gives the impression you don’t have a problem that you really do.

These systems are very simple and we can’t run any additional “client" software on it to “phone home” for service health and availability…something like Paessler's PRTG Network Monitor.

I did identify a few critical network services running on the systems and found that they communicated out on specific ports.

If we could run port-scans against those ports and found them open/listening, then that might provide a more accurate assessment of the servers’ health rather than the basic ping reply/no-reply feedback.

So here are a few of the tools and utilities I considered in that approach.

If you have any additional utilities or tricks for remotely monitoring server/service availability please drop a tip into the comment jar!

Cheers,

--Claus V.