A case of svchost.exe high CPU usage

Our developer sent me a message in our Lync Skype for Business and she told me that some of our customer apps are not responding. It seems that the back-end server (A Windows Server 2008 R2 with IIS and SQL 2012) is causing the lag. When I mean lag, I mean it is not responding. When I mean not responding, I mean it crashed.

So I asked our backend developers to check it out, they confirmed that The SQL Server process and full text died. Least of my worries, then MMC is not launching. Then you try it again and again, then this server shows everything that you opened earlier *toinks*. 

Lets just restart the server *grin* I really don’t have time for this *grin*. Server did not recover, had to ask the client to start the machine from their VM consoles. All went back to normal, well after a few hours.

Its already past 6PM and I really got a bad feeling about this. Worse, its not our server per se, its in a hosted facility somewhere, a lot of developers / groups had used this server from before and it’s the clients machine. Ouch.

Anyway, I still have to diagnose what is happening or tomorrow we have no server. I miss Azure, sigh. So opening up “MSTSC” and logging into the server:

1. First thing that you do in this situation is to open up your instrumentations, you can use task manager but I use Performance Monitor first so that I can see what is really happening to the entire server in detail . So on your run command, type PERFMON then press enter.

Go to Performance > Data Collector Sets > System > System Performance. Right click and start this baby up. Once finished you will have this report:

SVCHOSTSo as you can see, I have a busy CPU. At 100% utilization, something is running or should I say hogging the CPU of this server.

2. Now we use Task Manager:

SVCHOST2

You see a certain “SVCHOST.EXE” is running and is taking a lot of CPU from the server. Something is really odd in this server, its not using a lot of memory but look at the CPU, also check the description.

Hmmmm… The plot thickens…You can actually right click the process and then choose properties, and here is mine:

SVCHOST3

Okay so its not the real SVCHOST! I now have evidence that this is an actual malware! Copy that location, screenshot it. We will delete it later. Lets go to the details, maybe there’s more info there.

 SVCHOST4

None, so lets proceed.

3. Kill it before it lays eggs!!!

Okay, if you can stop yours great! Delete the file but mine wont die. Nothing is like good ol’fashioned manual malware killing I see.

Experience tells me I cant install an antivirus now and convenience suggest its much harder work to do it, downloading from subscription, running updates yada yada yada.

So download the Process Explorer from Technet: https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

SVCHOST5

Lets find it and check what it is doing using process explorer: Sort this list by CPU so that we know which SVC Host.

SVCHOST6

Go to the properties and navigate to each tab. On the networking “TCP/IP” tab I just found out that this connects to 77.72.133.157 and its using port 52324 and 7777. Checked it out it’s a “Cryptonyte mining pool”. I really don’t care what it is, I know its not for the use of the server so lets kill it.

SVCHOST7

Then refresh it a couple of times so that we can see if it is still there. Another tool that you can use is the commandline for tasklist.

4. To check if its still a running process using another tool, open an elevated command prompt and type tasklist /SVC /FI “IMAGENAME eq svchost.exe

SVCHOST9

Not running anymore? Run it a couple of times, just to be sure.

5. Delete it. AKA Zombie Rule #2

While it is not running, we can now delete the actual executable file. Told you you need the address / path / folder where it is. In my case its in the temp directory of the administrator. Which by the way should have been disabled thru policy, but I gotta check later, for now, we just need to delete this. In my case delete the entire temp folder. Deleting it should be our double tap.

So do a couple more checks, lets see if its still there. If not, try to restart and do a couple of checks including performance monitor. Then we can give it a good bill of health, hope I can now back to steam blizzard origin TFS. Nah who am I kidding – going back to outlook.

6. What we learned so far:

> Never install anything that is not needed in the server: Text editors, freewares, winrar, etc. If you had to edit, edit it outside of the server. If you need to zip it, use the built-in file compressor by Windows.

  • > Always turn on firewall: As you can see, if only necessary ports are opened, the malware wouldn’t be able to connect to any host.

> Never use the server for downloading: Shame on you if you do this, never, ever use the server to download anything. Even it is EF updates, no. You do this on your machine, not in the server

> Never put server outside of DMZ and in the internet: There’s a reason why we do not do this. Ever. Use hardware firewalls or internet delivery or content delivery appliances. Don’t put your server bare naked in the internet

> Harden your servers. Security is a must. Get qualified consultants if you cant do this yourself. Test your hardening, create policies.

> Never use the default “ADMINISTRATOR”: Create another administrator account and use a different less administrative account. Disable Administrator if possible.

> GPO: Always manage servers with Group Policies.

> Common Sense

 

7. Resources that we used:

MSTSC: https://technet.microsoft.com/en-us/library/cc753907.aspx

Process Monitor: https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

    > To the Systinternal Guys, up till today, I use this. Thank you.

Performance Monitor: https://technet.microsoft.com/en-us/library/cc749249.aspx

Data Collector Sets: https://technet.microsoft.com/en-us/library/cc722148.aspx

Tasklist: https://technet.microsoft.com/en-us/library/bb491010.aspx

Zombieland Rule #2 reference : http://www.zombielandrules.com/zombieland-rule-2-double-tap/

Windows Server 2008 R2 DHCP Error 1046–not authorized

Its late last night and I am doing my usual labs and I just rebuilt my entire lab using Server Core. It’s a temporary lab for a customer POC that I will ship out tomorrow and with that I have combined AD + DNS + DHCP role in one VM. So here’s the story:

  1. Successfully Installed AD and DNS, thru DCPromo /unattend:c:DCUnnattend.txt – good.
  2. Installed DHCP server role using my trusted OCsetup and it installed flawlessly.
  3. Used netsh exec to load my usual DHCP configuration, same one that I have been using so far on any server install that I have for labs. Great.
  4. Now its not giving any IP addresses to other client VM’s – now why?
    I did some troubleshooting and found an error on the event log: Error 1046! (yea used wevtutil and viewing it thru event viewer but that’s for another story Smile)
      So what is Event 1046? TechNet says:

    Event ID 1046 — DHCP General Availability
    Applies To: Windows Server 2008

    General availability of the Dynamic Host Configuration Protocol (DHCP) server refers to its ability to service clients. General availability depends on:

    1. Proper authorization of the DHCP server
    2. Presence of Active Directory Domain Services
    3. Successful loading of the DHCP dynamic-link libraries (DLLs)

    http://technet.microsoft.com/en-us/library/cc726914(WS.10).aspx

    Wait I say to myself, isn’t this server core installation is also the AD Server of this domain? Because I know for a fact that if DHCP role is installed in a computer running the AD also, it does not have to be Authorized!

    Okay, lets authorize it via CMD using this:

    netsh dhcp server serverID initiate auth

    or

    netsh dhcp server initiate auth

    Oh this will be a long night. So saved the server VM, snapshot, shutdown and then I tried it again now using a template that is not from Core (Windows Server 2008 R2 Full + AD + DNS installed from WDS, another story for later). Tried same steps and there, Then me going to the DHCP snap-in on the server manager, I confirmed that the DHCP server still appears unauthorized. Out of desperation, I Restarted my server and while doing so reading this: http://support.microsoft.com/kb/279908 Not very helpful though. Ok out of frustration, I authorized my DHCP on the MMC Snap-in and everything works fine!

    Having that, I suspect I am using the wrong command to initiate the authorization. Okay going back to basics, as one of my very very dark mentor before when I was still on the ISP business “RTFM”!

    So going to http://technet.microsoft.com/en-us/library/dd379483(WS.10).aspx

    image

    Wait, what?! Nah, lets see the complete manual of netsh dhcp here: http://technet.microsoft.com/en-us/library/bb490941.aspx

    image

    So lets try it out, shut down the Server Fulls and restored the Server Core VM’s fired up this command:

    Netsh DCHP add server <fqdn> 192.168.1.2

    and it works, its alive! its alive!

    Okay lesson learned, if you are installing DHCP role with ADDS + DNS role, make sure that you add your fqdn and ip to the list of authorized servers in active directory. Snap-in authorization does this for us (I think) but if you are now using and adopting Server Core for the entire enterprise make sure that you fire up that command. Thus a new entry in my step-by-step commands on deploying Windows Server.

    Now back to my servers! Cheers! And oh Good morning!

    Enabling Audit Events for Windows Firewall with Advanced Security

    If you are following the TechNet Article http://technet.microsoft.com/en-us/library/ff428143(WS.10).aspx you may notice that if you use :

    auditpol.exe /list /category:"Policy Change"  or any category, this throws an error 0x00000057 that the parameter is incorrect when used with Windows Server 2008 R2 and Windows 8 Beta (I have not checked with Vista and XP).

    When /get is used rather than /list, there is no error and it displays correctly. Full command used is as follows:

    auditpol.exe /get /category:"Policy Change"

    image

    Hyper-V Windows clients not using Dynamic Memory on fresh install

    1

    Ok if you encounter a freshly installed Windows 7 client and are not using dynamic memory on hyper-V even though you have already set it in options that it should – try upgrading the Integration Services.

    Just go to actions then choose  “Insert Integrations Services Setup Disk”.

    1b

    Then when autoplay comes, just run.

    1c

    Click ok on UAC

    1d

    Click okay.

    2

    While installing, grab a beer and walk away.

    3

    Restart when appropriate.

    4

    Will install some updates….

    5

    After booting.. WHALAHH!

    All clients are now running dynamic memory!

    ADFS_SP2010_DN

     

     

     

    This post is dedicated to the one that called me earlier while I am at work asking about this so I figured, lets just post this overdue draft! Good luck tomorrow!

    SharePoint 2010 Stand Alone Installation – Error on Step 2: Microsoft.SharePoint.SPException: User cannot be found

    A thing worth posting, Note to SELF: If you are installing SharePoint 2010 on a standalone configuration, make sure that the machine where you are installing MUST HAVE ACCESS TO THE DOMAIN CONTROLLER where it is a member of.

    For the one that needs more info, here is the stack trace from the PSDiagnosticsLog:

    Exception: Microsoft.SharePoint.SPException: User cannot be found.
       at Microsoft.SharePoint.Administration.SPSiteCollection.Add(SPContentDatabase database, SPSiteSubscription siteSubscription, String siteUrl, String title, String description, UInt32 nLCID, String webTemplate, String ownerLogin, String ownerName, String ownerEmail, String secondaryContactLogin, String secondaryContactName, String secondaryContactEmail, String quotaTemplate, String sscRootWebUrl, Boolean useHostHeaderAsSiteName)
       at Microsoft.SharePoint.Administration.SPSiteCollection.Add(SPSiteSubscription siteSubscription, String siteUrl, String title, String description, UInt32 nLCID, String webTemplate, String ownerLogin, String ownerName, String ownerEmail, String secondaryContactLogin, String secondaryContactName, String secondaryContactEmail, Boolean useHostHeaderAsSiteName)

    P.S. No screenshot for now, as I have fixed it and would not be possible to replicate, besides its 1AM already, geez!

    Windows 7 and Server 2008 R2 Service Pack 1

    As you may already knew, the Service Pack 1 was already released for Windows Server 2008 R2 and Windows 7 clients as well.

    I had a few discussions with the other IT people and asking me, why am I installing the SP1. Generally speaking, why I want to install from a disk that has the service pack in it already is because I wanted to keep my PC as updated as possible and instead of me doing an upgrade to a machine that I will update anyways, I’ll just start from an image that has the service pack – minus the head ache of updating one’s machine.

     

    On the server side, I had a chance before to demo one of its feature the “Dynamic Memory” over a TechNet session here in Makati and if you are looking into Virtualization, you may want that feature – and its free with the update, why not yea?

    More info at this TechNet link: http://technet.microsoft.com/en-us/library/ff817622(WS.10).aspx and here http://www.microsoft.com/oem/en/downloads/pages/windows_7_sp1.aspx

    “more” – a CMD command that has been remembered and should not be forgotten

    Today is one of those days when I remember the good old days that we did great things even without a mouse, just the hard keyboard and the black terminal, I was trained to configure SNMP, MRTG, <you name it>, with just that. For some reason mouse does not work today, if I do plug any mouse HP KVM goes ballistic. So this afternoon I was stuck in configuring a network adapter thru cmd or the command line. Not hard actually but if you do “IPCONFIG /ALL” on Windows Server 2008 R2 to check, the output will be a looooong list of adapters, as seen in the this screenshot: that’s Adapter # 22 and another isatap. 

    Without a mouse, this will be impossible to read, for one it would scroll too fast for the eye – another is that you can’t scroll up. But there’s a way a combination of piping and the more command. A little review for the “younger” admins nowadays.

    Piping or | – reads the output from one command and writes it to the input of another command. For redirection commands, you can read thru here: http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/redirection.mspx?mfr=true

    More command – which displays one screen of output at a time. Read thru here: http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/more.mspx?mfr=true

    This is some of the lessons I got from Mr. D. P. Nava (Hello sir Darth!), is to redirect the output of one command, thru piping, to another command. Then we now get creative, this time, our first command outputs this long strings and redirect (pipe) those output to a command that makes the output only show line per line (more).

    To do this, here is a screen capture:

    so thats <Command1> <pipe/redirect> <Command2>

    or in this particular scenario ipconfig /all | more

    Hitting return creates this output:

    As you may have noticed, even if the output of the command IPCONFIG /ALL is really long, but it stopped to a point where it can only be displayed by the CMD prompt. To scroll down one line down hit enter again. Hitting SPACE scrolls it to a whole page. Pressing “Q”  quits the more command will return you to your usual CMD prompt.

    More command is not just for displaying output of another command, you can use this to read text files etc. The syntax now is More <FileName>. Example:

    this has the output of:

    Same keyboard shortcuts applies.

    Check out the in command help from more by giving it a /? switch

    It’s really nice remembering these things when you needed them most!

    Free E-Book: Understanding Microsoft Virtualization Solutions

    Either its a coincidence or just luck, right this very moment I am working with HYPER-V (or hyper-v working a VM export-import thing, oh yea, lemme blog about that next time!) and was reading some guides on it. I searched for a topic over the internet and this came up! Its a free Ebook that can be downloaded here: http://download.microsoft.com/download/5/B/4/5B46A838-67BB-4F7C-92CB-EABCA285DFDD/693821ebook.pdf

    Inside the book are some what I can call, GEMS like this one:

    But what I am really after is the Import and Export of Virtual Machines in Hyper-V, found at Page 79.

    “You can use the Hyper-V Manager console to export a virtual machine from one Hyper-V
    server so that you can import it onto a different Hyper-V server. This import/export functionality
    allows you to migrate a virtual machine from one host computer to another using a
    process called Quick Migration.”

    I had to read this really AFTER doing server migrations. I already had the guides from TechNet but this book re-validates everything that I did and yup its the same thing – and that’s GOOD!

    That’s all for now, I’m just waiting for the filecopy to finish, I’m importing the last VM’s now 😛

    Server and Tools Philippines

    Hi, Just a quick post on what I found over a friends status message. Its a new site from the New Efficiency with Microsoft Server and Tools in the Philippines : http://www.microsoft.com/philippines/server/

    Do check it out, I’m just glad they did this for the Philippine market 🙂

    PS: Become a fan on Facebook: http://www.facebook.com/pages/Server-and-Tools-Philippines/328450804525 😀 Cheers!

    Free MS Press eBook download: Introducing Windows Server 2008 R2

    There’s a new MSPRESS ebook that is available for free download: Introducing Windows Server 2008 R2.

    Contents at a Glance
    Chapter 1 What’s New in Windows Server R2
    Chapter 2 Installation and Configuration: Adding R2 to Your World
    Chapter 3 Hyper-V: Scaling and Migrating Virtual Machines
    Chapter 4 Remote Desktop Services and VDI: Centralizing Desktop and Application Management
    Chapter 5 Active Directory: Improving and Automating Identity and Access
    Chapter 6 The File Services Role
    Chapter 7 IIS 7.5: Improving the Web Application Platform
    Chapter 8 DirectAccess and Network Policy Server
    Chapter 9 Other Features and Enhancements

    Get it free from at Microsoft Downloads