Tag Archives: RAM

How to measure Windows 7 Memory Usage

I did a memory usage test of Windows 7 64 bit with 4 GB RAM. Since Windows 7 takes advantage of more RAM, let’s see the result if we test with 8 GB RAM. If you are using 8 GB RAM for your VDI sizing, this test will be more applicable than the test with 4 GB RAM because Windows 7 will make use of the extra RAM. It is smart enough to utilised the hardware since it’s already given to it.

BTW, if what you want is Windows 2008, see this.

Both tests (4 GB and 8 GB) were conducted on physical desktops. I did it on physical so we know for sure there is no hypervisor impacting any reading here. I know VMware VMkernel will not impact the reading, but to assure some readers, I decided to eliminate that layer altogether. The 2 desktops are not identical but they have the same set of applications. The 8 GB desktop drives a 4K monitor, while the 4 GB desktop drives a Full HD monitor. I’m not sure if the 4K display impacts RAM, as I thought it uses the video RAM instead. One thing for sure, it’s much easier to review on 4K display!

In my system, since I used SSD, SuperFetch is disabled. This informative post provides details on SuperFetch. BTW, if you are using VDI, the Horizon 6 manual recommends disabling it. The reason is “By disabling the Windows prefetch and superfetch features, you can avoid generating prefetch files and the overhead associated with prefetch and superfetch operations. This action can reduce the growth of linked-clone machines and minimize IOPS on full virtual machines and linked clones”

With that, let’s dive to the test.

Screenshot #1

11 starting

  • I boot up the machine, and let it idle for a few minutes to ensure all start up programs have finished running. I want them to “settle down”…
  • Windows 7 makes use of all the physical RAM. The 8 GB desktop takes 1 GB more RAM compared with the 4 GB desktop. I do not have any applications running and have listed all the processes shown in Task Manager. Windows 7 takes up 3 GB right away.

Screenshot #2

13 Free did dropped to 0 for 1 second

  • I performed a similar test to the one I did on the 4 GB desktop. Essentially, I launched a lot of common apps, and opened lots of large files (>10 MB on average). For the video, I opened a 650 MB video.
  • I also forced PowerPoint to load all the slides, by going into slide sorter and made it draw all slides.
  • Naturally, CPU and Disk would spike, so I let them settle down first. My focus here is RAM.
  • The screenshot is taken after Windows 7 settles down. As you can see, CPU metrics have gone down for all processes. PowerPoint, Visio, Adobe, Word, etc. have gone down to 0%, as they are done opening files.
  • Surprisingly, Windows 7 still have 1.5 GB of free RAM. This tells me that 6.5 GB is comfortable.

Screenshot #3

13 Free did dropped to 0 for 1 second

  • Just in case you think 8 GB RAM is too much, you can easily hit it by opening more applications and files. In the screenshot above, the Free memory dropped to 61 MB. It actually touched 0 MB for a second. Windows did not seem to like the 0 value and would move some pages out.
  • The screenshot was again taken after CPU and Disk stabilised. The RAM is also stable around 6 GB. Windows “Used Physical Memory” graph at the top right does not take into account Standby Memory.
  • Should you take into account Standby? That seems debatable.

Screenshot #4

15 commit RAM also stable

  • There is another way to know if Windows needs more RAM. I recommend you read this excellent article by Ed Bott. The link to the screenshots is not working, which is one reason I recreated my own.
  • From the PerfMon, I can see that my Commit Limit is 16 GB. Commit Limit = Physical RAM + Virtual RAM. In my case, that’s 8 GB of RAM and 8 GB of PageFile.sys. So 16 GB is all I have.
  • % Committed is what is currently committed / Commit Limit. In my case, Windows commits 6.2 GB. The value is stable, so the % Committed is stable at 38%.
  • Should you use your Virtual RAM? There are different opinions on the Internet. I personally prefer what I recommend at the end of this blog.
  • BTW, the pagefile.sys is typically located in C:\ directory. It’s hidden, just like the file for hibernation. By default, Windows 7 automatically manages the pagefile. I notice the same behaviour in Windows 8.1. Both basically creates a pagefile the size of your physical RAM. Whether pagefile is good or bad, that is again debatable…. My experience is at 8 GB, you need it. Windows 7 performs poorer without it.
  • One thing for sure. Pagefile is not swap file. Swap file is used when Windows runs out of physical RAM. Pagefile is used proactively. Just because you’re seeing activity in pagefile.sys does not mean Windows is running out of RAM.

Screenshot #5

16 impact of closing all apps

  • To complete the test, I closed all applications. You can see in the top right corner, the chart changes in value in tandem as Windows closed the applications.
  • Interestingly, the usage did not go back to original. Windows is using ~4.6 GB of RAM. I think some applications do not actually leave the RAM. Skype, Chrome are examples of such applications. I have seen Chrome taking up >1 GB of RAM if you let it run for days.
  • BTW, Windows 7 keeps the pages in the Standby memory. It does not move it out after a few minutes. To me, this makes engineering sense. It is the same strategy adopted by ESXi VMkernel, which is why you see the Memory Consumed counter to be high.

Screenshot #6 (video)

22 normal playing does not result in growing RAM - it is steady with minimal page fault

  • I’m curious the impact of playing video on RAM. I played a 650 MB video. Surprisingly, it did not occupy 650 MB. In fact, it occupied only 360 MB. How I know is the Free memory dropped by 360 MB. I played the video in full size (1:1), not full screen, which explains why it did not occupy the full screen as it’s 4K display.
  • During the play, the memory counter did not slowly go up. It remains essentially the same, as you can see above.
  • I thought perhaps because I was simply watching the video normally (sequentially). So I jumped along the video, forcing it to play randomly. I would click toward the end, let it play, then immediately click somewhere at the beginning. It is a 27 minutes video, so I have plenty of timeline to click. It’s interesting to see that this random jump does not result in page fault. It is as if the entire video is already in memory. The counter Hard Faults/sec barely moved. Perhaps it was reading from disk directly?

Conclusion

  • If you want the best performance, use Total – Free as the sizing.
  • If you want a more cost effective solution, use Total – Free – Standby.  This would result in around 1-3 GB less RAM.
  • Let Windows manage the pagefile. This is the default setting anyway. I noticed a visibly slower performance even though Windows showing >1 GB of Free memory. In fact, Windows gave error message, and some applications crashed.
  • The % Committed metric should not hit 80%. Performance drops when it hits 90%, as if it’s a hard threshold used by Windows. If you use a pagefile, you will not hit this limit.
  • In general, I’d size Windows 7 between 4 – 8 GB of RAM, depending on the users. I’d use the following guidelines
    • 4 GB for light user
    • 8 GB for average
    • 12 GB for heavy. Yes, that’s 12 GB as I’ve seen my customers hit near 0 Free Memory, and he is just a “normal, average user”. He is an IT Manager.

Additional Resource

You might be wondering what those memory counters mean in Task Manager. Naveed Qadri explains it well here, so please read it.

  • Cached = Standby + Modified
  • Available = Standby + Free
  • Free = Free + Zero.

The difference between Cached and Available is Cached uses Modified, while Available uses Free. Available means exactly what the word means. It is the amount of physical memory immediately available for used.

Another article on Windows 7 memory management that I found useful was one written by Brandon Paddock. You can find it here. He wrote a program that proves that Commit can go up without In Use going up. I’m going to quote a sentence, so you can find it in his blog.

“Notice how my physical memory usage is unchanged, despite the fact that Commit has now increased by the full 2.3GB of that file."

Committed RAM can go beyond the physical RAM, as it takes into account pagefile.sys. The Commit Limit is typically 2x your physical RAM. In the example that Brandon gave:

"In fact, my commit value is now 6GB, even though I have only 4GB of physical memory and less than 3GB in use.”

Mark Russinovich explains in this technet blog something that you need to know. There is Reserved memory, and then there is committed memory. Some applications like to have its committed memory in 1 long contiguous block, so it reserves a large chunk up front. I can think of Databases and JVM in this example. This reserved memory does not actually store meaningful application data or executable. Only when the application commits the page that it becomes used. Mark explains that “when a process commits a region of virtual memory, the OS guarantees that it can maintain all the data the process stores in the memory either in physical memory or on disk”.

Notice the word “on disk“. Yes, that’s where the pagefile.sys comes in. Windows will use either the physical memory or the pagefile.sys.

Your ESXi Host needs more RAM. Really?

This blog post is adapted from my book, titled VMware vRealize Operations Performance and Capacity Management. It is published by Packt Publishing. You can buy it at Amazon or Packt.

I mentioned earlier that vRealize Operations does not simply regurgitate what vCenter has. Some of the vSphere-specific characteristics are not properly understood by traditional management tools. Partial understanding can lead to misunderstanding. vRealize Operations starts by fully understanding the unique behavior of vSphere, then simplifying it by consolidating and standardizing the counters. For example, vRealize Operations creates derived counters such as Contention and Workload, then applies them to CPU, RAM, disk, and network.

Let’s take a look at one example of how partial information can be misleading in a troubleshooting scenario. It is common for customers to invest in an ESXi host with plenty of RAM. I’ve seen hosts with 256 to 512 GB of RAM. One reason behind this is the way vCenter displays information. In the following screenshot, vCenter is giving me an alert. The host is running on high memory utilization. I’m not showing the other host, but you can see that it has a warning, as it is high too. The screenshots are all from vCenter 5.0 and vCenter Operations 5.7, but the behavior is still the same in vCenter 5.5 Update 2 and vRealize Operations 6.0.

I’m using vSphere 5.0 and vCenter Operations 5.x to show the screenshots as I want to provide an example of the point I stated in Chapter 1 of the book, which is the rapid change of vCloud Suite and VMware SDDC products. I have verified that the behaviour is still the same in vSphere 5.5 U2.

Picture1

The first step is to check if someone has modified the alarm by reducing the threshold. The next screenshot shows that utilization above 95% will trigger an alert, while utilization above 90% will trigger a warning. The threshold has to be breached by at least 5 minutes. The alarm is set to a suitably high configuration, so we will assume the alert is genuinely indicating a high utilization on the host.

Picture2

Let’s verify the memory utilization. I’m checking both the hosts as there are two of them in the cluster. Both are indeed high. The utilization for vmsgesxi006 has gone down in the time taken to review the Alarm Settings tab and move to this view, so both hosts are now in the Warning status.

Picture3

Now we will look at the vmsgesxi006 specification. From the following screenshot, we can see it has 32 GB of physical RAM, and RAM usage is 30747 MB. It is at 93.8%utilization.

Picture4

Since all the numbers shown in the preceding screenshot are refreshed within minutes, we need to check with a longer timeline to make sure this is not a one-time spike. So let’s check for the last 24 hours. The next screenshot shows that the utilization was indeed consistently high. For the entire 24-hour period, it has consistently been above 92.5%, and it hits 95% several times. So this ESXi host was indeed in need of more RAM.

Picture5

Deciding whether to add more RAM is complex; there are many factors to be considered. There will be downtime on the host, and you need to do it for every host in the cluster since you need to maintain a consistent build cluster-wide. There will be lots of vMotion as each host needs to be shutdown. Because the ESXi is highly utilized, I should increase the RAM significantly so that I can support more VMs or larger VMs. Buying bigger DIMMs may mean throwing away the existing DIMMs, as there are rules restricting the mixing of DIMMs. Mixing DIMMs also increases management complexity. The new DIMM may require a BIOS update, which may trigger a change request. Alternatively, the large DIMM may not be compatible with the existing host, in which case I have to buy a new box. So a RAM upgrade may trigger a host upgrade, which is a larger project.

Before jumping in to a procurement cycle to buy more RAM, let’s double-check our findings. It is important to ask what is the host used for? and who is using it?.

In this example scenario, we examined a lab environment, the VMware ASEAN lab. Let’s check out the memory utilization again, this time with the context in mind. The preceding graph shows high memory utilization over a 24-hour period, yet no one was using the lab in the early hours of the morning! I am aware of this as I am the lab administrator in the past 6+ years!

We will now turn to vCenter Operations for an alternative view. The following screenshot from vCenter Operations 5 tells a different story. CPU, RAM, disk, and network are all in the healthy range. Specifically for RAM, it has 97% utilization but 32% demand. Note that the Memory chart is divided into 2 parts. The upper one is at the ESXi level, while the lower one shows individual VMs in that host.

The upper part is in turn split into 2. The green rectangle (Demand) sits on top of a grey rectangle (Usage). The green rectangle shows a healthy figure at around 10 GB. The grey rectangle is much longer, almost filling the entire area.

The lower part shows the hypervisor and the VMs’ memory utilization. Each little green box represents 1 VM.

On the bottom left, note the KEY METRICS section. vCenter Operations 5 shows that Memory | Contention is 0%. This means none of the VMs running on the host is contending for memory. They are all being served well!

Picture6

I shared earlier that the behavior remains the same in vCenter 5.5. So, let’s take a look at how memory utilization is shown in vCenter 5.5.

The next screenshot shows the counters provided by vCenter 5.5. This is from a different ESXi host, as I want to provide you with a second example. It is not related to the 1st example at all. This host has 48 GB of RAM. About 26 GB has been mapped to VM or VMkernel, which is shown by the Consumed counter (the highest line in the chart; notice that the value is almost constant). The Usage counter shows 52% because it takes from Consumed. The active memory is a lot lower, as you can see from the line at the bottom. Notice that the line is not a simple straight line, as the value goes up and down. This proves that the Usage counter is actually the Consumed counter.

vSphere 5.5 ESXi memory counters

Notice that the ballooning is 0, so there is no memory pressure for this host. Can you explain why? It is because Consumed is below the threshold.

At this point, some readers might wonder whether that’s a bug in vCenter. No, it is not. ESXi is simply taking full advantage of the extra RAM. This is a similar behaviour by Windows 7 and Windows 2008.

There are situations in which you want to use the consumed memory and not the active memory. In fact, some applications may not run properly if you use active memory. I will cover this when we discuss memory counters in further chapters. Also, technically, it is not a bug as the data it gives is correct. It is just that additional data will give a more complete picture since we are at the ESXi level and not at the VM level. vRealize Operations distinguishes between the active memory and consumed memory and provides both types of data. vCenter uses the Consumed counter for utilization for the ESXi host.

As you will see later in this book, vCenter uses the Active counter for utilization for VM. So the Usage counter has a different formula in vCenter depending upon the object. This makes sense as they are at different levels. In my book I explain these 2 levels.

vRealize Operations uses the Active counter for utilization. Just because a physical DIMM on the motherboard is mapped to a virtual DIMM in the VM, it does not mean it is used (read or write). You can use that DIMM for other VMs and you will not incur (for practical purposes) performance degradation. It is common for Microsoft Windows to initialize pages upon boot with zeroes, but never use them subsequently.

Having said all the above, none of the counters & information above should be used in isolation. You should always check the VMs. It’s a common mistake to look at ESXi RAM counters (be it Consumed, Active, Usage, or Workload) and conclude there is RAM performance or RAM capacity issue. You should always check the VMs first. Are they having Memory Contention? If not, there is no RAM performance issue. What about Capacity? It’s the same principle. Take into account performance first, before you look at utilization. See this series of articles on what super metric you need.

For further information on this topic, I would recommend reviewing Kit Colbert’s presentation on Memory in vSphere at VMworld, 2012. The content is still relevant for vSphere 5.x. The title is Understanding Virtualized Memory Performance Management and the session ID is INF-VSP1729. If the link has changed, here is the link to the full list of VMworld 2012 sessions.

Not all performance management tools understand this vCenter-specific characteristic. They would have given you a recommendation to buy more RAM.

Now that you know your ESXi does not need so much RAM, you can consider a single socket ESXi.

[20 March 2016: added 3rd example]

User Jengl at VMware communities posted a question here. He wanted to know why, when “we patched the half of our cluster we had high consumed/usage of memory and the vCenter started to initiate balloon, compress and finally swap on the ESXi Hosts. I was not expecting that, because active memory was only a small percentage of consumed.

He shared his screenshot, which I copied here so I can explain it.

pastedImage_0

In the above screenshot, it looks like ESX Host (and not cluster or VM). We could see around 10:50 am, there was a spike. The Memory Usage counter hit 98.92%. That’s high, as that’s an average of 5 minutes. It probably touched 100% within that 5 minutes. Memory Workload, which is based on Active, was low at 19%. This is possible. The VMs were not actively using the RAM. Contention rose to a 1.82%, which indicates some VMs were contending for RAM.

The whole situation went back to normal. Contention disappeared pretty quickly, and Memory Usage went down. It’s hard to see fro the chart, but it looks like it dropped below 95%.

Based on the above, I’d expect balloon to have kicked in at 10:50 am too. It would then taper off. Because the VMs were not actively using the RAM, I’d not expect the value to drop to 0.

Guess what? The screenshot below confirmed that.

pastedImage_2

I hope that’s clear.

Right-sizing Windows 7 RAM

Right-sizing RAM is certainly a common requirement, as we know VM tend to be oversized. However, we need to be careful when reducing RAM. Reducing RAM on VM based on information at the hypervisor level can result in poor performance. You need to have in-guest level data.

I did a set of test on a physical desktop to see Windows memory behaviour. I did it on physical so we know for sure there is no hypervisor impacting any reading here.

I have 64 bit Windows 7 on a physical desktop with 4 GB RAM. For a test on 8 GB RAM, see this as the result is not the same, even though the workload is similar.

Screenshot #1

1 - Win 7 - after boot - stable - 0 page fault

  1. I boot up the machine, and let it idle for a few minutes to ensure all start up programs have finished running. I want them to “settle down”…
  2. Modified (orange), based on this excellent article, means memory that can be used by other programs, but Windows needs to write to the page file before they can be reused. If that is true, I wonder why it is called modified.
  3. The screenshot shows that Windows 7 64 bit uses 1296 MB of RAM (162 + 1055 + 79). The Standby Memory means it contains cached data that is no longer actively in used. How does Windows define “actively in used” is not clear. If a block of RAM is not used in the past 1 minute, is it active? How about 10 minutes? 1 hour? The next few screenshots will clarify that.
  4. We have 674 MB of Standby and 2126 of Free as you can see above.

Screenshot #2

2 - Win 7 - after opening lots of files and apps - CPU

  1. I then launched a lot of common apps, and open lots of files. I opened 13 large PowerPoint files and 10 Adobe PDF files. You can see in the screenshot, the task bar is full of documents.
  2. Naturally, CPU and Disk would spike, so I let them settle down first. My focus here is RAM.
  3. The screenshot is taken after Windows settles down.
  4. You notice CPU has gone down. My desktop has 4 cores, and 2 cores are actually parked. You can see that the only CPU-consuming process is the Resource & Performance Monitor itself, and the Desktop Window Manager, as it has to redraw the screen.
  5. Both PowerPoint and Adobe has gone down to 0%, as they are done opening files.

Screenshot #3

3 - Win 7 - after opening lots of files and apps - RAM stable

  1. After verifying that CPU has settled down, I verify the RAM.
  2. Unlike CPU, we have a very different situation here for RAM. This is key to explaining why you cannot just downsize RAM without consulting application team.
  3. Windows now use 2553 MB. Much higher than 1296. Obviously the executables of PowerPoint, Adobe, Excel are in the RAM.
  4. The Free Memory has gone down from 2126 to just 469 MB.
  5. The Standby Memory increases from 674 to 1074. This is the 23 ppt/pdf files that were opened and after that not touched by. All I did was just opened them. So from here we can tell that “active” is a relatively short time, it’s in seconds or minutes. In my subsequent observation, it is actually in seconds.

Screenshot #4

4 - Win 7 - after forcing powerpoint to draw all slides by going into slide sorter view and scroll - same with adobe - RAM increase

  1. Based on screenshot #3 (not #4), I could conclude whether PowerPoint or Adobe read the entire files. I think it reads more than the first page or the first slide, but probably not all. I knew from other tests that 1 MB of powerpoint file translates to 15 MB in RAM. I still have 469 MB of Free RAM, while I think it should be lower.
  2. So I forced PowerPoint to read the entire content of almost every files. What I did what go to each file, change to “Slide Sorter” view, wait until PowerPoint redraws all slides, scroll down, wait until it redraws, repeat until the last slide is drawn. This ensures PowerPoint read every slide, and brought it to RAM. I did the same thing some of the Adobe file.
  3. The Free RAM dropped from 469 MB to just 9 MB. It’s interesting it does not go down to 0. This is showing Windows memory management. The Standby remains high, from 1074 to 953. This is a proof that “active” is a relatively short time.

Screenshot #5

5 - Win 7 - after idling for 40 minutes - RAM is still high because there is no pressure to release

  1. I let my desktop idle for 40 minutes. Naturally, Windows went to sleep mode after a while. I wanted to see the effect of sleep mode on RAM.
  2. When I brought up Windows from Sleep mode, I checked the RAM. Screenshot #5 shows that it’s hardly change. The Free RAM remains low, Standby is about the same, and In Use is about the same.
  3. It looks like Windows treat executables differently than it does data. I think the In Use Memory is the apps, and not the data. However, I’m not 100% sure here.
  4. It is confirmed that Sleep does not flush the RAM content.
  5. I am not able to check page fault as Resource Monitor does not have it. But when I use PerfMon, I can see paging.
  6. You can see paging at vCenter if you separate the page file into its own vmdk file. This is something you should consider as standard. You can then use vRealize Operations to monitor for abnormal behaviour. For example, the VM owner complains that the VM became slower since last weekend change. You do not see any significant difference in memory, CPU, Disk or network utilisation. Your ESXi host was healthy too, delivering very low contention. However, vRealize Operations shows abnormal behaviour in the vmdk file that hosts the page file (and only the page file). It has much higher read and write. From here you can at least say that there is excessive paging. This topic is covered in depth here.

Screenshot #6

6 - Win 7 - after closing all apps - RAM is cleared

  1. I closed all apps. As expected, the memory got back to near the original situation, when I had no apps open.

Screenshot #7

7 - Win 7 - after launching PowerPoint and open 8 MB file

  1. I launch PowerPoint again, but this time only open 1 file (8 MB). I do not launch other apps.
  2. Notice the Standby Memory remains the same. So Windows keeps it, and uses the Free Memory instead. This makes sense.

Screenshot #8

8 - Win 7 - Standby and Free do not reduce even if we use more than 4 GB

  1. I launched Visio installer and OpenOffice installer. Total 560 MB file.
  2. I also launched Word, Adobe, etc.
  3. I notice Windows Free Memory did touch 0 MB for 1 second, and then Windows keeps it above it. You notice in the screenshot that the Standby Memory actually goes up by around 500 MB.
  4. This confirms that Standby is adjusted in seconds, not minutes. This means Active is in seconds too.

Screenshot #9

9 - Win 7 - Closing installer software removes 500 MB from RAM

  1. I closed both the installer programs.
  2. The Active memory (In Use) remains the same.
  3. The Free Memory goes up by around 500 MB, while Standby drops by 500 MB.
  4. This means we know the installer was in the Standby Memory.

Screenshot #10

10 - Win 7 - Closing 25 MB of ppt files frees up 380 MB

  1. I closed 25 MB worth of PowerPoint files.
  2. The Standby remains the same, so we know the files were not in Standby. They were in “In Use”, as it drops and the Free Memory goes up by 380 MB.

Conclusion:

  1. Memory tends to stay even though it is not used. It is much more stable than CPU. This is because Windows does not know when it will be used again, so it just keeps the pages there, just in case. This makes sense. So the RAM remains idle.
  2. Because they are idle, the Hypervisor (VMkernel) will think they are not used. Remember the hypervisor has no visibility inside the Guest OS memory list. The Memory Usage counter in vCenter will give you a low number. If you are sizing based on this, it can impact performance, as it can force paging inside Windows.
  3. For Windows, you need to use the Free Memory counter, In Use counter, and Standby counter to give a more accurate picture. For optimum performance, I’d include Standby in the sizing as that’s serving as cache