Memory

Problems caused by memory shortages can often appear to be problems in other parts of the system. You should monitor memory first to verify that your server has enough, and then move on to other components. To run Windows 2000 and IIS 5.0, the minimum amount of RAM a dedicated Web server needs is 128MB, but 256 MB to 1GB is often better. Additional memory is particularly beneficial to e-commerce sites, sites with a lot of content, and sites that experience a high volume of traffic. Since the IIS File Cache is set to use up to half of available memory by default, the more memory you have, the larger the IIS File Cache can be.

Note Windows 2000 Advanced Server can support up to 8GB of RAM and Windows 2000 DataCenter can support up to 64GB, but the IIS File Cache cannot take full advantage of this. If you add the /3GB switch in c:\boot.ini, then inetinfo.exe can address up to 3GB of memory; otherwise, it's limited to 2GB of address space. In addition, every instance of dllhost.exe can address up to 2GB, so if you had a sufficiently large system, the cumulative memory used by all the IIS-related processes could go beyond 4GB. As ISAPIs and ASPs execute at medium isolation by default, they are in an instance of dllhost separate from inetinfo.exe. In addition, high-isolation applications each have their own dllhost.

The static file cache lives in the inetinfo process and can store up to 1.5GB of content¹ . The ASP caches live in each process hosting asp.dll. The caches draw upon the total memory available to the process hosting them.

To determine if the current amount of memory on your server will be sufficient for your needs, use the Performance tool (formerly known as PerfMon) that is built in to Windows 2000. The System Monitor, which is part of the Performance tool, graphically displays counter readings as they change over time.

Also, keep an eye on your cache settings—adding memory alone won't necessarily solve performance problems. You need to be aware of IIS cache settings and how they affect your server's performance. If these settings are inappropriate for the loads placed on your server, they, rather than a lack of memory, may cause performance bottlenecks. For more information about these cache settings, see the IIS Settings section and Appendix 1: Performance Settings of this document. For a discussion about caching with ASP and IIS, see Appendix 3: ASP Caching.

Note When using Performance counters to monitor performance, you can see a description of any counter by selecting that counter in the Add Counters dialog and clicking Explain.

Log the following counters to determine if there are performance bottlenecks associated with memory:

• Memory: Available Bytes. Try to reserve at least ten percent of memory available for peak use. Keep in mind that IIS 5.0 uses up to 50 percent of available memory for its file cache by default.
• Memory: Page Faults/sec, Memory: Pages Input/sec, Memory: Page Reads/sec, and Memory: Transition Faults/sec. If a process requests a page in memory and the system cannot find it at the requested location, this constitutes a page fault. If the page is elsewhere in memory, the fault is called a soft page fault (measured by Transition Faults/sec). If the page must be retrieved from disk, the fault is called a hard page fault. Most processors can handle large numbers of soft faults without consequence. However, hard faults can cause significant delays. Page Faults/sec is the overall rate at which the processor handles faulted pages, including both hard and soft page faults. Pages Input/sec is the total number of pages read from disk to resolve hard page faults. Page Reads/sec is the number of times the disk was read to resolve hard page faults. Pages Input/sec will be greater than or equal to Page Reads/sec and can give you a good idea of your hard page fault rate. If these numbers are low, your server should be responding to requests quickly. If they are high, it may be because you've dedicated too much memory to the caches, not leaving enough memory for the rest of the system. You may need to increase the amount of RAM on your server, though lowering cache sizes can also be effective.
• Memory: Cache Bytes, Internet Information Services Global: File Cache Hits %, Internet Information Services Global: File Cache Flushes, and Internet Information Services Global: File Cache Hits. The first counter, Memory: Cache Bytes, reveals the size of the File System Cache, which is set to use up to 50 percent of available physical memory by default. Since IIS automatically trims the cache if it is running out of memory, keep an eye on the direction in which this counter trends. The second counter is the ratio of cache hits to total cache requests and reflects how well the settings for the IIS File Cache are working. For a site largely made up of static files, 80 percent or more cache hits is considered a good number. Compare logs for the last two counters, IIS Global: File Cache Flushes and IIS Global: File Cache Hits, to determine if you are flushing objects out of your cache at an appropriate rate. If flushes are occurring too quickly, objects may be flushed from cache more often than they need to be. If flushes are occurring too slowly, memory may be wasted. See the ObjectCacheTTL, MemCacheSize, and MaxCachedFileSize objects in Appendix 1: Performance Settings.
• Page File Bytes: Total. This counter reflects the size of the paging file(s) on the system. The larger the paging file, the more memory the system commits to it. Windows 2000 itself creates a paging file on the system drive; you can create a paging file on each logical disk, and you can change the sizes of the existing files. In fact, striping a paging file across separate physical drives improves paging file performance (use drives that do not contain your site's content or log files). Remember that the paging file on the system drive should be at least twice the size of physical memory, so that the system can write the entire contents of RAM to disk if a crash occurs.
• Memory: Pool Paged Bytes, Memory: Pool Nonpaged Bytes, Process (inetinfo): Virtual Bytes, Process (dllhost#n) : Virtual Bytes, Process (inetinfo): Working Set, and Process (dllhost#n): Working Set.. Memory: Pool Paged Bytes and Memory: Pool Nonpaged Bytes monitor the pool space for all processes on the server. The Virtual Bytes counters monitor the amount of virtual address space reserved directly by IIS 5.0, either by the Inetinfo process (in which the core of IIS runs) or by the Dllhost processes (in which isolated or pooled applications run) instantiated on your server. The Working Set counters measure the number of memory pages used by each process. Be sure that you monitor counters for all instances of Dllhost² on your server; otherwise, you will not get an accurate reading of pool space used by IIS. The system's memory pools hold objects created and used by applications and the operating system. The contents of the memory pools are accessible only in privileged mode. That is, only the kernel of the operating system can directly use the memory pools; user processes cannot. On servers running IIS 5.0, threads that service connections are stored in the nonpaged pool along with other objects used by the service, such as file handles and sockets.

Besides adding more RAM, try the following techniques to enhance memory performance: improve data organization, try disk mirroring or striping, replace CGI applications with ISAPI or ASP applications, enlarge paging files, retime the IIS File Cache, eliminate unnecessary features, and change the balance of the File System Cache to the IIS 5.0 Working Set. The last of these techniques is detailed later in this document.

For a detailed discussion list of Windows 2000 and IIS 5.0 settings that will affect these counter numbers, see Appendix 1: Performance Settings.

Processor Capacity

With users demanding quick response time from Web sites and the increasing amount of dynamically generated content on these sites, a premium is placed on fast and efficient processor usage. Bottlenecks occur when one or more processes consume practically all of the processor time. This forces threads that are ready to be executed to wait in a queue for processor time. Adding other hardware, whether memory, disks or network connections, to try to overcome a processor bottleneck will not be effective and will frequently only make matters worse.

IIS 5.0 on Windows 2000 Server scales effectively across two to four processors, and with a little additional tuning scales well on eight processors (see Appendix 4). Consider the business needs of your Web sites if you're thinking of adding more processors. For example, if you host primarily static content on your server, a two-processor computer is likely to be sufficient. If you host dynamically generated content, a four-processor setup may solve your problems. However, if the workload on your site is sufficiently CPU-intensive, no single computer will be able to keep up with requests. If this is the case for your site, you should scale it out across multiple servers. If you already run your site on multiple servers, consider adding more.

You should be aware, however, that the biggest performance gains with Windows 2000 and IIS 5.0 result from resolving inadequate memory issues. Before you make any decisions about changing the number of processors on your Web servers, rule out memory problems and then monitor the following Performance Counters.

• System: Processor Queue Length. This counter displays the number of threads waiting to be executed in the queue that is shared by all processors on the system. If this counter has a sustained value of two or more threads, you have a processor bottleneck on your hands.
• Processor: %Processor Time. Processor bottlenecks are characterized by situations in which Processor: % Processor Time numbers are high while the network adapter card and disk I/O remain well below capacity. On a multi-processor computer, it's a good idea to examine the Processor: % Processor Time counter to pick up any imbalance.
• Thread (Inetinfo/thread-number): Context Switches/sec, Thread (Dllhost/thread-number#process-instance): Context Switches/sec, and System: Context Switches/sec. If you decide to increase the size of any of the thread pools, you should monitor the three counters listed here. Increasing the number of threads may increase the number of context switches to the point where performance decreases instead of increases. Ten context switches or more per request³ is quite high; if these numbers appear, consider reducing thread pool size. Balancing threads against overall performance as measured by connections and requests can be difficult. Any time you tune threads, follow-up with overall performance monitoring to see if performance increases or decreases. To determine if you should adjust the thread count, compare the number of threads and the processor time for each thread in the process to the total processor time. If the threads are constantly busy, but are not fully using the processor time, performance may benefit from creating more threads. However, if all the threads are busy and the processors are close to their maximum capacity, you are better off distributing the load across more servers rather than increasing the number of threads. See also the AspThreadGateEnabled and AspProcessorThreadMax metabase properties in Appendix 1: Performance Settings in this document.
• Processor: Interrupts/sec and Processor: % DPC Time. Use these counters to determine how much time the processor is spending on interrupts and deferred procedure calls (DPCs). These two factors can be another source of load on the processor. Client requests can be a major source of each. Some new network adapter cards include interrupt moderation, which accumulates interrupts in a buffer when the level of interrupts becomes too high.

Scaling Out Across Multiple Computers

If processor problems persist, try scaling your site out across multiple computers using Network Load Balancing (NLB) or a hardware load balancer. While setting up a Web farm using one of these methods adds a layer of complexity and introduces a number of other issues, this action is likely to solve a number of your performance issues if your site is large enough. For more information about NLB, see the Network Load Balancing Technical Overview .

Network Capacity, Latency, and Bandwidth

Essentially, the network is the line through which clients send requests to your server. The time it takes for those requests and responses to travel to and from your server is one of the largest limiting factors in user-perceived server performance. This request-response cycle time is called latency, and latency is almost exclusively out of your control as a Web server administrator. For example, there is little you can do about a slow router on the Internet, or the physical distance between a client and your server.

On a site consisting primarily of static content, network bandwidth is the most likely source of a performance bottleneck. Even a fairly modest server can completely saturate a T3 connection (45Mbps) or a 100Mbps Fast Ethernet connection. You can mitigate some of these issues by tuning the connection you have to the network and maximizing your effective bandwidth as best you can.

The simplest way to measure effective bandwidth is to determine the rate at which your server sends and receives data. There are a number of Performance counters that measure data transmission in many components of your server. These include counters on the Web, FTP, and STMP services, the TCP object, the IP object, and the Network Interface object. Each of these reflects different Open System Interconnectivity (OSI) layers. For a detailed list of these counters and their analysis, see the Internet Information Services 5.0 Resource Guide, released with the Windows 2000 Server Resource Kit. In particular, see the Network I/O section of the Monitoring and Tuning Your Server chapter. To start, however, use the following counters:

• Network Interface: Bytes Total/sec. To determine if your network connection is creating a bottleneck, compare the Network Interface: Bytes Total/sec counter to the total bandwidth of your network adapter card. To allow headroom for spikes in traffic, you should usually be using no more than 50 percent of capacity. If this number is very close to the capacity of the connection, and processor and memory use are moderate, then the connection may well be a problem.
• Web Service: Maximum Connections and Web Service: Total Connection Attempts. If you are running other services on the computer that also use the network connection, you should monitor the Web Service: Maximum Connections and Web Service: Total Connection Attempts counters to see if your Web server can use as much of the connection as it needs. Remember to compare these numbers to memory and processor usage figures so that you can be sure that the connection is the problem, not one of the other components.

See the Tuning and Troubleshooting Suggestions section of this document for suggestions on how to reduce bandwidth usage by reducing file sizes and by enabling proxy and client caching.

Disk Optimization

Since IIS 5.0 writes logs to disk, there is regular disk activity even with 100 percent client cache hits. Generally speaking, if there is high disk read or write activity other than logging, this means that other areas of your system need to be tuned. For example, hard page faults cause large amounts of disk activity, but they are indicative of insufficient RAM.

Accessing memory is faster than disk seeks by a factor of roughly 1 million (nanoseconds versus milliseconds); clearly, searching the hard disk to fill requests will degrade performance. The type of site you host can have a significant impact on the frequency of disk seeks. If your site has a very large file set that is accessed randomly, if the files on your site tend to be very large, or if you have a very small amount of RAM, then IIS is unable to maintain copies of the files in RAM for faster access.

Typically, you will use the Physical Disk counters to watch for spikes in the number of disk reads when your server is busy. If you have enough RAM, most connections will result in cache hits unless you have a database stored on the same server, and clients are making dissimilar queries. This situation precludes caching. Be aware that logging can also cause disk bottlenecks. If there are no obvious disk-intensive issues on your server, but you see lots of disk activity anyway, you should check the amount of RAM on your server immediately to make sure you have enough memory.

To determine the frequency of disk access, log the following counters:

• Processor: % Processor Time, Network Interface Connection: Bytes Total/sec, and PhysicalDisk: % Disk Time. If all three of these counters have high values, then the hard disk is not causing a bottleneck for your site. However, if the % Disk Time is high and the processor and network connection are not saturated, then the hard disk may be creating a bottleneck. If the Physical Disk performance counters are not enabled on your server, open a command line and use the diskperf -yd command.

See also the Tuning and Troubleshooting Suggestions section.

Security

Balancing performance with users' concerns about the security of your Web applications is one of the most important issues you will face, particularly if you run an e-commerce Web site. Since secure Web communication requires more resources than non-secure Web communications, it is important that you know when to use various security techniques, such as the SSL protocol or IP address checking, and when not to use them. For example, your home page or a search results page most likely doesn't need to be run through SSL. However, when a user goes to a checkout or purchase page, you will want to make sure that page is secure.

If you do use SSL, be aware that establishing the initial connection is five times as expensive as reconnecting using security information in the SSL session cache. The default timeout for the SSL session cache has been changed from two minutes in Windows NT 4.0 to five minutes in Windows 2000. Once this data is flushed, the client and server must establish a completely new connection. If you plan on supporting long SSL sessions, consider lengthening this timeout with the ServerCacheTime registry setting (described in Appendix 1). If you expect thousands of users to connect to your site using SSL, a safer approach is to estimate how long you expect SSL sessions to last, then set the ServerCacheTime parameter to slightly longer than your estimate. Do not set the timeout much longer than this or else your server may leave stale data in the cache. Also, make sure that HTTP Keep-Alives are enabled (on by default). SSL sessions do not expire when used in conjunction with HTTP Keep-Alives unless the browser explicitly closes the connection.

In addition to all security techniques having performance costs, Windows 2000 and IIS 5.0 security services are integrated into a number of operating system services. This means that you can't monitor security features separately from other aspects of those services. Instead, the most common way to measure security overhead is to run tests comparing server performance with and without a security feature. The tests should be run with fixed workloads and a fixed server configuration, so that the security feature is the only variable. During the tests, you probably want to measure the following:

• Processor Activity and the Processor Queue: Authentication, IP address checking, SSL protocol, and encryption schemes are security features that require significant processing. You are likely to see increased processor activity, both in privileged and user mode, and an increase in the rate of context switches and interrupts. If the processors in the server are not sufficient to handle the increased load, queues are likely to develop. Custom hardware, such as cryptographic accelerators, may help here.

  If the SSL protocol is being used, lsass.exe may consume a surprising amount of CPU. This is because SSL processing occurs here. This means that administrators used to monitoring CPU usage in Windows NT may see less processor consumed by Inetinfo.exe and more consumed by Isass.exe.

• Physical Memory Used: Security requires that the system store and retrieve more user information.
• Network Traffic: You are also likely to see an increase in traffic between the IIS 5.0-based server and the domain controller used for authenticating logon passwords and verifying IP addresses.
• Latency and Delays: The most obvious performance degradation resulting from complex security features like SSL is the time and effort involved in encryption and decryption, both of which use lots of processor cycles. Downloading files from servers using the SSL protocol can be 10 to 100 times slower than from servers that are not using SSL.

If a server is used both for running IIS 5.0 and as a domain controller, the proportion of processor use, memory, and network and disk activity consumed by domain services is likely to increase the load on these resources significantly. The increased activity can be enough to prevent IIS 5.0 services from running efficiently. It is highly recommended that you refrain from running a high-traffic webserver on a domain controller.

Monitoring Your Web Applications

Upgrading a poorly written application to one that is well designed and has been thoroughly tested can improve performance dramatically (sometimes as much as thirty fold). Keep in mind, however, that your Web applications may be affected by back-end latencies (for example, legacy systems such as AS/400). Remote data sources may cause performance problems for any number of reasons. If developers design applications to get data from another Web site, and that Web site crashes, it can cause a bottleneck on your server. If applications are accessing a remote SQL Server database, the database may have problems keeping up with requests sent to it. While you may be the administrator of your site's SQL database, it can be difficult to monitor these servers if they are remotely located. Worse, you may have no control over the database servers, or other back end servers. If you can, monitor the back-end servers that work with your applications and keep them as well tuned as you do your Web server.

To determine if your Web applications are creating a bottleneck on your server, monitor the following performance counters:

• Active Server Pages: Requests/Sec, Active Server Pages: Requests Executing, Active Server Pages: Request Wait Time, Active Server Pages: Request Execution Time, and Active Server Pages: Requests Queued. If you are running ASP applications on your server, these counters can provide you with a picture of how well the applications are performing. Active Server Pages: Requests/Sec does not include requests for static files or other dynamic content and will fluctuate considerably based on the complexity of the ASP pages and the capacity of your Web server. If this counter is low during spikes in traffic on your server, your applications may be causing a bottleneck. Requests Executing indicates the number of requests currently executing (i.e., number of active worker threads); Request Wait Time indicates the number of milliseconds the most recent request was waiting in the queue, and Request Execution Time indicates how many milliseconds the most recent request took to execute. Ideally, Requests Queued and Request Wait time should remain close to zero, but they will go up and down under varying loads. The maximum number for Requests Queued is determined by the metabase setting for AspRequestQueueMax. If the limit is reached, client browsers will display "HTTP 500/ ServerToo Busy." If these numbers deviate a great deal from their expected range, your ASP applications will likely need to be rewritten to improve performance. Request Execution Time can be somewhat misleading because it is not an average. For example, if you regularly receive 30 requests for a page that executes in 10 milliseconds (ms) to every one request for a 500ms page, the counter is likely to indicate 10ms, although the average execution time is over 25ms. It's hard to say what's a good value for Requests Executing. If pages execute quickly and don't wait for I/O (loading a file or making a database query), this number is likely to be low (little more than the number of processors when the machine is busy). If pages must wait for I/O, the number of pages executing is likely to be higher (anything up to AspProcessorThreadMax multiplied by the number of processors multiplied by the number of processes hosting ASP). If Requests Executing is high, Requests Queued is large, and the CPU utilization is low, you may need to increase AspProcessorThreadMax. When enabled, thread gating seeks to optimize Requests Executing (see the IIS Settings section of this document). The user's response time is proportional to Request Wait Time plus Request Execution Time plus network latency. Note: these counters are the sum of the ASP performance counters for each process hosting ASP, and there is no way to break them down by process.
• Web Service: CGI Requests/sec and Web Service: ISAPI Extension Requests/Sec report the rates at which your server is processing CGI and ISAPI application requests. If these values drop while under increasing loads, you may need to have the application developers revisit their code.

  Note ASP is an ISAPI Extension and is included by the second counter.

• Web Service: Get Requests/sec and Web Service: Post Requests/Sec reflect the rate at which these two common HTTP request types are being made to your server. POST requests are generally used for forms and are sent to ISAPIs (including ASP) or CGIs. GET requests account for almost all other requests from browsers and include requests for static files, requests for ASPs and other ISAPIs, and CGI requests.

Tuning Your Web Applications

IIS 5.0 is typically very efficient at serving static HTML pages with the out-of-the-box settings. If your site hosts primarily static content, many performance problems can be hardware related. IIS 5.0 offers improved performance for Web applications, but some additional tuning may be required to optimize performance. Of course, issues about best practices in Web application design and coding remain, regardless of improvements in server software. While this document does not attempt to discuss the intricacies of tuning your Web applications, this section does provide some pointers and recommendations to make them perform faster. Consider the following while planning and testing your Web applications before running them on your production servers.

First of all, ISAPI applications run faster than Active Server Pages (ASP) applications, though there is much less developer overhead for ASP. Both of these types of applications run faster than equivalent CGI applications.

Second, you should use static files wherever you can because they don't have the processing load or cause the disk activity that dynamic files do. In conjunction with this, your applications should push as much of the processing load onto the client as possible in order to avoid network latencies. This also saves on server-side resources and allows changes to appear instantaneously to the user. A common example is adding client-side code to validate that forms have been filled out with good data, such as checking that e-mail addresses are well formed or credit card numbers have a correct checksum.

Another tactic is to make sure that debugging for ASP is turned off on your production servers. If debugging is enabled, you will need to set the AppAllowDebugging metabase property to FALSE. For more information, see Appendix 1: Performance Settings.

Set Expires headers for all images and for HTML wherever possible to allow both to be stored in the client's cache. For more information, see the Tuning and Troubleshooting Suggestions section of this document.

Do not store Apartment-threaded components in ASP Application and Session state. This includes all Microsoft Visual Basic® components, but not Java or most C++ objects.

Use Secure Sockets Layer (SSL) only when necessary. Using the HTTPS protocol is much more expensive than standard HTTP. Be sure that the information being sent, such as credit card numbers or medical information, is sensitive enough to warrant the added expense. For more information about security tuning issues, see the Security section of this document.

Process isolation will affect Web application performance as well. IIS 5.0 Web applications run in the out-of-process pool (medium protection) by default. It is safer to take the performance impact of process isolation than to risk server downtime and data loss that can be caused by a low-isolation application crashing the Inetinfo process. For a more in-depth discussion of this topic, see the Process Isolation section of this document.

To enhance database-driven performance in a production environment, use Microsoft SQL Server 2000. Because both IIS and SQL Server perform best with plenty of memory, try storing the database on a separate server from the Web service. In this situation, communication across computer boundaries is frequently faster than communication on a single computer. Performance degradation due to a lack of memory and insufficient cycles often occurs when both SQL Server and IIS reside on the same server. Also, be sure to create and maintain good indexes. This will minimize I/O on your database queries. Last but not least, take advantage of stored procedures. They take much less time to execute and are easier to write than an ASP script designed to do the same task.

As a general rule of thumb, if you have an ASP script that's more than 100 lines long (counting lines of code in files brought in using the #include directive), consider creating a COM+ component to provide the same function. If written efficiently and debugged properly, COM+ components can offer twenty to thirty times the processing speed of a script for the same dynamic page. The easiest way to measure the size of an ASP script with #includes is to change the file extension of the page from .asp to .stm and open the .stm file with your browser. Use your browser's View Source command to display the .asp file and lines of code from the included files.

For you to maximize performance for your dynamic Web applications, it is very important to stress test your applications before you make them live on your site. If your production webserver is a multiprocessor system, it is important to stress test on a multiprocessor system. This will help identify multiprocessor scaling problems and race conditions in your script and components. A very good tool for doing this is the Web Application Stress (WAS) tool, which can be downloaded from the Microsoft Web Application Stress Tool site (http://webtool.rte.microsoft.com/). Included at this site are a tutorial and a knowledge base dedicated to the tool. WAS is also included on the Windows 2000 Resource Kit companion CD.

For more information about tools for measuring the performance of your Web servers and applications, see the Tools to Monitor and Test Server Performance section of this document. For a list of links and references about Web application performance and the tools to test that performance, see the Resources section of this document.

Tools to Monitor and Test Server Performance

To support your performance tuning and testing needs, Microsoft offers a number of tools: some included with Windows 2000 and IIS 5.0, others offered on the Windows 2000 Resource Kit CD, and still others downloadable from the Microsoft Web site. The System Monitor (formerly known as PerfMon) is built in to Windows 2000 and is essential to monitoring nearly every aspect of server performance. Process and Thread Status (pstat.exe) shows the status of all running processes and threads. Process Tree (ptree.exe) allows you to query the process inheritance tree and kill processes on local or remote computers. These tools are available on the Windows 2000 Server Resource Kit companion CD. The HTTP Monitoring Tool, available on the companion CD to the Windows 2000 Resource Kit, monitors HTTP activity on your servers and can notify you if there are changes in the amount of activity. Network Monitor is a Windows 2000 administrative tool you can use to keep tabs on network traffic. It is not installed by default, but you can install it by using the Add/Remove Programs feature of the Control Panel.

Note a lightweight version of Network Monitor comes with Windows 2000 Server; the full-featured version is available with Microsoft Systems Management Server. NetStat is a command line tool that detects information about your server's current network connections.

At the center of these tools are the Performance Counters that are built into IIS 5.0 and the Windows 2000 operating system. Developers can also include custom Performance Counters in the ISAPI DLLs or COM components that they write. These counters can be read directly by a number of the tools mentioned above, including System Monitor, the Web Application Stress Tool, and WCAT. A number of these counters have been mentioned throughout this document; it is important to know which are pertinent to your monitoring and testing requirements.

System Monitor is the single most important tool to establish a baseline of performance on your Web server and monitor the effects on performance of any changes you make to software or hardware. System Monitor provides a UI that allows you to see performance counter readings whether you are monitoring or logging them. It also allows you to graphically log counter activity and set alerts that will appear in Event Viewer. System Monitor provides documentation for each counter in your system.

The Web Application Stress tool is designed specifically to simulate multiple browsers requesting pages from a Web site. You can use this tool to gather information about the performance and stability of your Web applications and about how your servers are performing. This tool simulates a large number of requests with a relatively small number of client machines. The goal is to create an environment as similar to a production environment as possible. This allows you to find and eliminate problems in your Web server and applications prior to deploying them on your production servers.

For more information on any of these tools, see the online IIS 5.0 documentation included in the Windows 2000 Resource Kit. Links to other sources of information are included in the Resources section of this document.

Features and Settings in Windows 2000 and IIS 5.0

If you currently have a well-tuned Web site running on Windows NT Server 4.0 with IIS 4.0, that site should perform well on Windows 2000 Server and IIS 5.0. For more information, see the white paper, Windows 2000 Performance: An Overview ( http://www.microsoft.com/windows2000/guide/platform/performance/overview.asp). You will want to monitor your server and site as you make the transition. You should be aware of some new features in Windows 2000 and IIS 5.0 that are designed for better performance and ease of administration. In addition, there are some changes in default settings in IIS 4.0 to IIS 5.0. This section discusses these features and changes.

Setting Windows 2000 as an Application Server

If you plan to use your server primarily as a Web server, setting up your server computer as an application server is a quick way to improve performance. This allows you to take advantage of better SMP scalability, improved networking performance, and support for more physical memory for your Web applications. If you run COM-based applications, using Windows 2000 as an application server also takes better advantage of improvements to COM+. In addition, you can use the transaction-processing capabilities of COM+ as a transaction monitor to improve performance of database applications. Windows 2000 Server installs as a file server by default, so you should make sure to select the application server during the installation process. If you don't, however, it is easy to configure your server as an application server after installation. To do so:

1. Click Start, point to Settings, and click Network and Dial-up Connections.
2. Select Local Area Connection and open its properties.
3. Select File and Printer Sharing for Microsoft Networks and open its properties.
4. On the Server Optimization tab, select Maximize data throughput for network applications.

This will not take effect until you reboot the server.

IISReset Utility

IIS 5.0 offers a number of new features and default settings to help make Web sites that run on it more reliable and easier to administer. The first of these is the new IISReset.exe, a utility that allows you to stop and restart IIS services without rebooting your computer. By default, IISReset will restart your services if they fail. You can also use IISReset to remotely start, stop, or pause your services, and to reboot your server computer if necessary. You should reboot only as a last resort. If you restart your Web service with IISReset, users will experience a small pause, during which they have only to hit refresh to get a new page. If the entire computer is rebooted, the unavailability is longer. You can also isolate the services that you stop. For example, if you are running an SMTP server on the same computer as your Web server, you can choose to simply stop and restart your Web service, rather than taking down the SMTP services as well.

You should be aware that frequent reboots and resets will compromise the integrity of your performance data. If you are using IISReset to automatically restart services, it may mask this problem, so you should always monitor the Event Log for restarts.

IIS Settings

The AspProcessorThreadMax metabase property has changed. Formerly called ProcessorThreadMax and stored in the registry in IIS 4.0, its default value was 10. The new default value in IIS 5.0 is 25. This setting is per processor and per process: on a dual-processor system: the number of worker threads in each process can be up to twice as high as the AspProcessorThreadMax value, or up to 50 worker threads (with the default settings). If you are running several high-isolation ASP applications, each process will have an independent set of worker threads.

Note ASP starts out with a number of worker threads that is equal to the number of processors plus seven. It creates more threads when the size of the ASP request queue passes certain thresholds.

The AspThreadGateEnabled property has been added to the metabase. It is off by default. If you turn this property on, IIS performs thread gating which dynamically controls the number of concurrently executing threads in response to varying load conditions. When processor utilization drops below 50 percent, which could indicate that threads are blocked (for example, while waiting for an external database to return the results of a query) or simply that the load is light, IIS 5.0 increases the number of active threads so that other requests can be serviced in a timely manner. When processor utilization exceeds 80 percent, indicating a heavy load, IIS 5.0 deactivates threads to reduce the amount of context switching. Both lower and upper limits can be set: AspThreadGateLoadLow defaults to 50 percent, while AspThreadGateLoadHigh defaults to 80 percent. Regardless of the value of AspThreadGateEnabled, an ASP process will never have more worker threads than the number of processors multiplied by AspProcessorThreadMax.

For sites that do a lot of ASP processing, it is best to test performance with thread gating turned on and with it turned off, to see what the effects are. Make your final decision based on your observations. For sites that are made up primarily of static files, turn the setting on and monitor server performance to see if throughput and response time are improved.

IIS 5.0 has also changed the default behavior of the ASP Template Cache. In IIS 4.0, the ASP Template Cache limit defaulted to –1. With this setting, this cache could grow arbitrarily large. On Web sites with lots of ASP content, the ASP Template Cache tended to fill all of the RAM in the server. In contrast, the IIS 5.0 default limit is 250 files. Because each site has its own requirements, you should reset the limit to meet your site's particular needs. Perhaps the easiest way to accomplish this is to monitor performance as you increase and decrease the value. Because an entry in this cache can be pointed to by one or more entries in the ASP Script Engine Cache, and because best performance occurs if the scripts in ASP pages are found in the ASP Script Engine Cache, you should never set the limit on the ASP Template Cache to zero. Doing so prevents any hits on the ASP Script Engine Cache, because the ASP Script Engine Cache entry for a particular .asp file can only be referenced through its template. Thus, if no templates are cached, the ASP Script Engine Cache is rendered useless. ASP Script Engine Cache hits provide better performance than hits on the ASP Template Cache, so if you make ASP Script Engine Cache hits impossible, performance suffers badly unless all your pages are static. From IIS 4.0 to IIS 5.0, the ASP Script Engine Cache limit has been upped from 30 to 125 files. To determine if you need to change your cache settings, you should keep an eye on response times, the number of ASP requests in the queue, the number of context switches, and the amount of CPU utilization. See also Appendixes 3 and 4.

Note The ASP Script Engine Cache setting should be at least equal to one more than the number of CPUs on your server, multiplied by the AspProcessorThreadMax setting. It is probably too small for most 8P systems.

Also, you should consider adjusting the default settings for the IIS File Cache. You can add these settings to the registry to modify IIS 5.0 default behavior. The first setting you should consider adding is the MemCacheSize object; if this is not present in the registry, the default behavior is to allow the cache to grow to a maximum of half the available physical memory. This ensures that IIS interacts well with other applications on machines that are not dedicated Web servers. Try upping this limit (specified in MB) and monitoring performance to see if there are any gains. The second registry object you should consider adding is MaxCachedFileSize. The IIS default behavior is to allow a maximum file size of 256KB in the cache. If you have a site that has several large JPEG files that are accessed regularly, you may want to experiment with bumping this limit up to determine if caching files larger than 256KB will work for your site. But be aware that if file sizes are around 200 to 300KB, you'll reach a point of diminishing returns when caching them. For smaller files, the overhead of reading from disk rather than the IIS File Cache is significant. For larger files, you won't get much performance improvement; you're more likely to just waste memory. IIS regularly purges from the cache files that have not been requested recently (within the last 30 seconds, by default). The threshold is determined by the ObjectCacheTTL (TTL stands for Time To Live) registry setting; by default this is not present in the registry. If you have plenty of memory, it may be effective to adjust this TTL upwards.

For a discussion of how IIS and ASP use caches to process incoming requests, see Appendix 3: ASP Caching.

Process Isolation

IIS 4.0 introduced the concept of running Web applications out of process. This feature created greater stability for Web servers, but at a significant performance cost. In IIS 5.0, the performance of out-of-process applications has improved, especially for ASP. Some performance degradation remains, however, in comparison with IIS 5.0 in-process applications. In addition to improved performance, the concept of running applications out of process has been expanded. You can now run Web applications in a pooled out-of-process environment.

Applications that run in the Web services process (Inetinfo.exe) result in higher performance, but there is a greater risk that a misbehaving application can make the Web services become unavailable. The recommended configuration is to run Inetinfo.exe in its own process, run mission-critical applications in their own processes (high protection), and run remaining applications in a shared, pooled process (medium protection). For the best performance and reliability, run ASP applications in medium protection and configure any COM+ components as library applications, not server applications.

If you decide to run your application as a separate process, or with other applications in a single pooled process, you will need to select High (Isolated) or Medium (Pooled) from the Application Protection drop-down list on the Home Directory or Virtual Directory property sheet. You should first create an application directory and designate it as either a Home Directory or Virtual Directory, if you haven't already done so. By default, all new applications are run in medium protection. You can run a very large number of applications at medium isolation, but you will only be able to run a few dozen applications at high isolation, because each process consumes significant resources.

For more information about these registry settings and metabase properties, see Appendix 1: Performance Settings. For more information about the features mentioned in this section, see the IIS 5.0 and Windows 2000 online documentation.

Tuning and Troubleshooting Suggestions

If you determine that you need to address specific hardware-driven performance issues, consider using the following suggestions.

• Upgrade to Larger L2 Caches. If you determine that you need to add or upgrade processors, choose processors with a large secondary (L2) cache. Server applications, such as IIS, benefit from a large processor cache because their instruction paths involve many different components and they need to access a lot of data. A large processor cache (2 MB or more if it is external, up to the maximum available if it is on the CPU chip) is recommended to improve performance on active servers running IIS 5.0.
• Upgrade to Faster CPUs. Web applications particularly benefit from faster processors.
• Set Aggressive Connection Timeouts. To combat network latency as much as you can, set aggressive connection timeouts. This is particularly important if you run a high traffic Web site. Open connections degrade performance. The ConnectionTimeout metabase property is set to 15 minutes by default. For more information on this property see Appendix 1: Performance Settings.
• Use Expires Headers. Set Expires headers on both static and dynamic content to allow both types of content to be stored in the client's cache. This makes for faster response times, places less load on the server, and less traffic on the network. For example, you could create a header that specifies not to download your company's logo .jpg file if the user has already visited your site. To set Expires headers for static content, use the HTTP Headers property sheet. To set Expires headers for ASP pages, use the Response.AddHeader method. For more information on this method, see the IIS 5.0 online documentation.
• Make Sure That ASP Buffering is Enabled. ASP buffering is on by default after a clean install of Windows 2000. If you have upgraded from Windows NT 4.0 and IIS 4.0, you may need to turn it on. ASP buffering allows all output from the application to be collected in the buffer before being sent across the network to the client browser. This cuts down on network traffic and response times. Although buffering reduces response times, it may leave users with the perception that the page is slower and less interactive, as they see no data until the page has finished executing. Judicious use of Response.Flush can improve the perception of interactivity. For more information about the Response.Flush method, see the IIS 5.0 online documentation and the discussion in the ASP Tips article listed in Appendix 5: Resources. See also the AspBufferingOn metabase entry in Appendix 1: Performance Settings.
• Lengthen Connection Queues and Use HTTP Keep-Alives. If you determine that your server does not have adequate bandwidth to meet demand and you are planning for increasing request loads, you can make better use of network bandwidth by doing two things: lengthening connection queues and verifying that HTTP Keep-Alives are enabled.

  Each IIS 5.0 service (FTP, Web, etc) has a connection queue, which is set to 15 entries. If this number, under load, does not meet your needs, you can increase it by adding a ListenBackLog parameter to the registry and setting the value to the maximum number of connection requests you want the server to maintain. For more information, see Appendix 1: Performance Settings.

  HTTP Keep-Alives maintain a client's connection to the server even after the initial request is complete. This feature reduces latency, reduces CPU processing, and optimizes bandwidth. HTTP Keep-Alives are enabled by default. To reset them if they have been disabled, select a site in the Internet Services Manager, open the site's Properties sheet, click the Performance tab, and select the HTTP Keep-Alives check box.

• Reduce File Sizes. You can increase the performance of your web server by reducing the sizes of the files being served. Image files should be stored in an appropriate compressed format. Cropping images and reducing color depths will also reduce file sizes. Limit the number of images and other large files where possible. You can also reduce file size by tightening up HTML and ASP code. Remove redundant blocks of code from ASP pages and make sure your HTML documents are efficiently authored. In particular, good use of Cascading Style Sheets can reduce the amount of HTML needed in individual pages.
• Store Log Files on Separate Disks and Remove Nonessential Information. If your server hosts multiple sites a separate log file is created for each site; disk writes for these logs can create a bottleneck on your server. Try storing logs on a separate partition or disk from your Web server. Another way to reduce disk bottlenecks is to avoid logging non-vital information. For example, you could place all of your image files in a virtual directory (for example, /images) and disable logging for that directory. To do this, open the property sheet of the directory, clear the Log visits check box, and then click OK. You could also use scripts or ISAPI filters to do this pruning. If yours is a particularly busy or large site, this can save you up to several gigabytes of disk space per day and significant log post-processing time.
• Use RAID and Striping. To improve disk access, use a redundant array of independent drives (RAID) and striped disk sets. You may also want to consider a drive controller with a large RAM cache. If your site relies on frequent database access, move the database to another computer.
• Defragment your Disks. Access times grow longer as disks become more fragmented. Windows 2000 comes with a Disk Defragmenter.
• Use CPU Throttling If Necessary. IIS 5.0 introduces two new features that deal with rogue applications: process accounting, which logs the CPU and other resources used by a Web site, and process throttling, which limits the amount of resources a Web site can consume.

Process accounting and process throttling work for both CGI (Common Gateway Interface) applications and for applications that are run out of process. You cannot activate accounting for in-process applications or for applications run in the new IIS 5.0 out-of-process pool (medium protection).

To turn on process accounting

1. In the Internet Services Manager, select the Web site that you want to set up process accounting on.
2. Open the site's property sheet and click the Home Directory tab.
3. In the Application Settings box, select High (Isolated).
4. On the site's property sheet, click the Web Site tab, and make sure Enable Logging is selected.
5. On the Web Site property sheet, click the Logging Properties button, and select Process Accounting.

The first two steps set the Web site to run out of process and the last two steps activate process accounting for that site.

For example, if you are an ISP and one of your customer sites is using more than its share of CPU time, you can activate process accounting and extend logging so that figures for the Job Object counters are recorded. With the information gathered from process accounting, you can then decide whether to upgrade the servers in your installation, to adjust the billing for this particular customer, or to limit the amount of resources the site can consume.

After determining the amount of resources the customer's site is consuming, you might want to limit that customer to a certain percentage of your available resources. This will free up resources for other customers. To limit a site's resources, run the site's applications out of process, and then turn on process throttling as follows:

1. On the site's property sheet, click the Performance tab.
2. Select Enable process throttling.
3. In the Maximum CPU use box, set the percentage of CPU resources dedicated to the site.
4. Select Enforce limits.

When the site reaches a predefined limit, it will take the defined action, such as reducing process priority, halting processes, or halting the site. Be aware that the site may actually exceed the apparent processor use limit if virtual directories within a throttled site are configured as in-process or pooled-process applications. In-process and pooled-process applications are not affected by processor throttling and are not included in process accounting statistics.

• The following techniques will help you determine if you need to use processor throttling: log the Processor: % Processor Time, Web Service: Maximum CGI Requests, and Web Service: Total CGI Requests counters; enable process accounting so that Job Object counters are included in IIS logs; and examine the Dllhost object counters to determine the number of out-of-process WAM and ISAPI requests.

You should keep in mind that process throttling could sometimes backfire. Because the throttled Dllhost process is running at a lower priority, it won't respond quickly to requests from the Inetinfo process. This can tie up several I/O threads, harming your server's overall responsiveness. As always after making any kind of change, monitor your server closely once you have set up process throttling to see what effects it has on performance.

Testing, Piloting, and Going Live

Before you deploy Windows 2000 with IIS 5.0 on your Web server computers, it is very important that you test your proposed design in an environment that simulates real-world scenarios as closely as possible. Not only does this help you find problems with your servers and the Web applications that you plan to deploy on them, but also it protects your production servers from being randomized by unpredictable problems. It is best if you set up your tests in a controlled environment, such as a lab, and isolate the servers from extraneous loads. Concentrate the test servers on stressing your hardware setup and Web applications.

Testing plays a vital role in the success of your upgrade from IIS 4.0 to IIS 5.0. In your test environment, you could discover a number of types of problems that would be disastrous to encounter on your live site. These might include any number of issues that will affect your Web server's performance. You may discover that you need to add more RAM, or that the ASP application you were hoping to deploy along with your upgrade of IIS 5.0 has too many bugs to go on the Web. If you eliminate as many of these issues as possible during the testing stage, you will have a greater chance of a smooth upgrade.

One recommended course of action is to use a methodical upgrade from IIS 4.0 to IIS 5.0. This involves creating a set of tests for IIS 4.0, running them, then performing the upgrades and running the exact same tests on IIS 5.0. Not only does this allow you to find any performance-related problems, but it also allows you to estimate the performance gains you will receive from upgrading. You can use tools like System Monitor and the Web Application Stress tool to monitor performance during tests and to generate test scenarios, respectively.

Once testing is complete, it is suggested that you set up an IIS 5.0 system. This means going live with your servers to a limited audience who will help you stress test your servers and applications in an environment more closely resembling real-world conditions than your test lab. Using your company's intranet could be an optimal environment to pilot your new deployment. During the pilot, you test your design in a controlled real-world environment in which users perform their normal business tasks using the new features. Remember to continue monitoring the performance of your servers throughout the pilot period. For details about designing your Test and Pilot deployments, see the Windows 2000 Deployment Planning Guide .

While both testing and piloting are excellent practices, neither can completely reproduce the type of use and amount of load that your Web servers will experience. After all, testing and piloting occur in controlled environments where network latency is minimal, and the kind and amount of requests being generated are known. When your servers and applications go live, you expose them to the entire Internet and its users.

It is vital that you continue to monitor your servers after the deployment of IIS 5.0 onto the production computers is complete. As stated earlier in this document, this will allow you to establish baseline performance records that you can use to determine whether performance is going up or down. Don't forget to compare the baseline numbers to the new numbers any time you make a change to your production servers, so that you can see what effects your change has had on performance. It is best if you only make one change at a time; otherwise, you will often not be able to tell which change produced which effect. If you make multiple changes, it is likely you won't be able to determine the effects of either. If performance does not improve as you expected, continue to analyze the data and make adjustments as indicated. Monitoring should continue on a regular basis, but tweaking performance settings will become less necessary over time.

Appendix 1: Performance Settings

You can adjust both IIS metabase properties and registry settings to tune the performance of your Web server. If you do plan to make changes to the registry, do not use a registry editor unless you have no alternative. Registry editors bypass the standard safeguards provided by administrative tools. These safeguards prevent you from entering conflicting settings or settings that are likely to degrade performance or damage your system. Editing the registry directly can have serious, unexpected consequences that can prevent the system from starting and require you to reinstall Windows 2000. The same can be said for making changes to the IIS metabase with the adsutil administrative utility (found in the Inetpub\AdminScripts directory). To configure or customize Windows 2000 and IIS 5.0, use the programs in Control Panel or Microsoft Management Console (MMC) whenever possible.

Metabase Settings

This list includes the most important metabase settings for tuning your Web servers. These can be retrieved and changed using the ADSI interfaces. Most of these settings will not take effect until the Web service is restarted. Refer to the section on the IISReset utility for more information.

AppAllowDebugging—This property specifies whether ASP debugging is enabled on the server. If this is the case, IIS application threads will be serialized, meaning that only one thread at a time will be allowed to execute for each application. This will adversely affect Web server performance. You should set this property to FALSE (the default) on all production servers.

AspAllowSessionState—The default value is TRUE. Turning this to FALSE can lead to better performance. Once changed, developers will have to explicitly override this setting in pages that need to make use of the Session object. To change the default setting for a single page, developers can use <% @EnableSessionState=False %> at the top of the page. Be sure to notify developers if you change this setting.

AspBufferingOn—The default value is TRUE, after a clean install of Windows 2000. If the machine was upgraded from NT 4, you may need to turn it on. This property's default behavior allows all output from an application to be collected in the ASP output buffer before the buffer is flushed to the client browser. If this property is set to FALSE, output from ASP scripts will be written to the client browser as it becomes available. You should make sure this property is set to TRUE on all production Web servers. For more details, see the Tuning and Troubleshooting Suggestions section of this document.

AspThreadGateEnabled (default value is FALSE) and AspProcessorThreadMax (default value is 25)— Thread gating is enabled when you set AspThreadGateEnabled to TRUE, and IIS 5.0 dynamically changes the number of worker threads in response to changing workloads. The maximum number of worker threads that IIS will allow per ASP process is AspProcessorThreadMax multiplied by the number of processors on your server. Reduce this value and monitor performance. If performance degrades, bring the AspProcessorThreadMax value back up. For more information, see the Thread Gating section of this document.

AspRequestQueueMax—The default limit for requests in the queue has been increased to 3,000 for IIS 5.0. The effectiveness of this setting depends on the behavior of the application. If the request's execution time is very short and the time in the queue will be short, it is reasonable to increase this limit.

AspQueueConnectionTestTime—This is a new setting for IIS 5.0 and aids significantly in the performance of Web applications. In IIS 4.0, execution of a request was unconditionally started when it was removed from the queue. In IIS 5.0, if a request has been in the queue longer than the queue connection test time, the server checks to see that the client is still connected before beginning execution. This feature handles the problem of impatient users filling up the request queue with numerous attempts at the same page. The default value is three seconds. Base your decision to change this value on the type of Web application your server is running. Long-running ASP pages can also make use of the Response.IsClientConnected method to see if the client is still waiting for the rest of the page. Long-running pages should make judicious use of Response.Flush to ensure that users perceive that the page is still alive and doing productive work. For more information about the Response object methods, see the IIS 5.0 online documentation.

AspSessionMax and AspSessionTimeout—The default behavior is to limit the length of single sessions to twenty minutes rather than to limit the number of concurrent sessions. For applications that take advantage of sessions, it may be prudent to reduce the Session Timeout to reduce the overhead required of the server, but if concurrent sessions increase to unwieldy proportions, it may be necessary to include a Session Maximum.

AspScriptEngineCacheMax—The new default for the maximum number of script engines to cache in memory is 125. This does not include currently running script engines. Adjust this according to the type of content in the application. If there are thousands of unique pages, there is probably some gain associated with increasing the cache size so that the most frequently requested pages can be readily accessed. A hit in the script engine cache means that you can avoid recompiling the template into byte code.

Before you set this metabase property, you should understand how ASP uses the ASP Script Engine Cache and the ASP Template Cache. For further discussion, see Appendix 3: ASP Caching.

AspScriptFileCacheSize—This property specifies the number of precompiled script files to store in the ASP Template Cache. If 0, no script files will be cached. If -1, all script files requested will be cached. The default value is 250. Increase this value if you have many different ASP pages. Do not set the value of this property to 0. This turns off all ASP caching and will severely impair your server's performance.

AspTrackThreadingModel—This metabase property specifies whether IIS will check the threading model of any components that your application instantiates. If this metabase property is left set to its default value (FALSE), the overhead incurred by ASP's threading model tracking is avoided, and you might see performance improvements in your ASP application. However, if this property is set to FALSE, any components that you create that you plan on giving Application scope must be agile: either marked as Both-threaded and aggregate the free-threaded marshaller or marked as ThreadingModel=Neutral. If an Application-scoped component is not agile, ASP will generate an error when you try to instantiate the component. In addition, if this property is FALSE, any objects that lack OnStartPage or OnEndPage methods and are instantiated in your ASP application will be released earlier than they would otherwise. This should improve your application's scalability. The default value for this property in IIS 4.0 was TRUE. It is not recommended that you enable it in IIS 5.0. For more information, see the IIS 5.0 online documentation.

CacheISAPI—This property indicates whether ISAPI extensions are cached in memory after use. If the value of this property is TRUE, the DLL file will remain in the cache until the server is stopped. If the value is FALSE, ISAPI extensions are unloaded from memory once the extension DLL is no longer in use. ISAPI extensions are cached or not cached based on the value of this property at the time they were loaded into memory for use. Thus, if this property is changed after an extension has been loaded and cached, the change will have no effect on that extension.

Setting this property to FALSE can be helpful for debugging, but make sure that it is set to TRUE on all production Web servers. Reloading an ISAPI extension DLL file for every request is very expensive and will degrade performance. ASP.dll is itself an ISAPI extension, so disabling this property will also degrade ASP performance.

ConnectionTimeout—This property specifies the time in seconds the server will wait before disconnecting an inactive connection. The default value is 900 seconds (15 minutes). Since open connections can degrade performance, consider lowering this value and monitoring your servers to see what impact this change makes.

MaxEndpointConnections—This property specifies the maximum number of "listen" sockets that will be aggregated on a network endpoint. For example, if this value is set to 15, a maximum of 15 total connections can be made to a single port, even if more than one domain is bound to the port. The lower of this property value and the value of the ServerListenBackLog property determines the number of sockets pooled on your server. Set this to a high number and monitor performance. The default value is 100.

ServerListenBackLog—This property specifies the number of outstanding sockets that can be queued. The lower of this property value and the value of the MaxEndpointConnections property determines the number of sockets pooled on your server. Set this to a high number and monitor performance. The default value is based on the AcceptEx operating system parameter and on the server size specified in the ServerSize metabase property. If ServerSize is set to 0, the default for this property is 5. If ServerSize is 1, the default is 40. If ServerSize is 2, the default is 100. Valid values for this property range from 5 to 1000.

ServerSize—This property specifies the general size of the server, in terms of number of client requests processed per day. A value of 0 indicates a small Web site that would expect to receive fewer than 10,000 requests per day, a value of 1 indicates a medium site handling between 10,000 and 100,000 requests a day, and a value of 2 designates a large site processing more than 100,000 requests a day. The default value is 1 (medium). To maximize the number of requests your server can handle, set this property to 2. You can adjust this setting using the UI. Open your site's property sheet, select the Performance tab, and adjust the Performance tuning slider to More than 100,000.

Registry Settings

This section lists the registry settings that you should be most concerned with when you tune your Web server. Included is the registry path for settings that reside in the same location, along with the name, range, default value and a description of what each setting does. You will have to restart Web services on your server for new Inetinfo settings to go into effect. For more information, see the IISReset Utility section of this document.

Registry Path:
HKEY_LOCAL_MACHINE \SYSTEM
\CurrentControlSet
\Services
\Inetinfo
\Parameters
DisableMemoryCache REG_DWORD

Range: 0, 1

Default: 0

Make sure that this parameter is set to 0 on all production servers. If this parameter is set to 1, static file caching will be disabled. While this may be useful during debugging, disabling caching can severely compromise production server performance. This parameter cannot be configured by using the IIS snap-in.

MaxCachedFileSize REG_DWORD

Range: 0 – unlimited (measured in bytes)

Default: 262,144 bytes (256KB) if value is not in registry.

This parameter determines the maximum size of a file that can be placed in the cache. IIS will not cache files that are larger than MaxCachedFileSize bytes. If you are running large dedicated Web servers, you may want to add this value to the registry to increase the file size that the cache can hold.

MemCacheSize REG_DWORD

Range: 0 MB – Total MB of Available RAM

Default: 50% of available memory if value not in registry

This parameter specifies the maximum amount of memory that IIS will use for its file cache. If IIS does not need this much memory, it will be left for other applications to use. If this value is not in the registry, IIS will use no more than half of the available memory on the Web server (which is dynamically calculated every 60 seconds). If you are running large dedicated Web servers, you may want to add this value to the registry and increase the amount of memory that IIS can use. You must specify this size in MB when you add this object to the registry.

ObjectCacheTTL REG_DWORD

Range: 0 - Unlimited

Default: 30 seconds

This parameter controls the Time To Live (TTL) setting of the static file cache, which defines the length of time that objects, including files, are held in cache memory. If an object in the memory cache has not been referenced for the defined period, that object will be phased out of the cache. By default, this value is not included in the registry. If you wish to change it, you must add it manually. If system memory is limited, or the server's contents are dynamic, you can use a lower TTL to prevent system memory from being used to cache a large number of volatile objects. Setting the value to 0xFFFFFFFF disables the object-cache scavenger and allows cached objects to remain in the cache until they are overwritten. Disabling the cache scavenger is useful if your server has ample system memory and your data is relatively static. Other sites may prefer the compromise of raising this value to several minutes.

PoolThreadLimit REG_DWORD

Range: 0 – Unlimited

Default: 2 * # MB

This parameter specifies the maximum number of I/O worker threads that can be created in the Inetinfo process, which limits the number of simultaneous connections. IIS sets PoolThreadLimit to 2 * number of megabytes of RAM present in the machine. If this value is larger than 256, it will be clamped to 256. If a value is present in the registry, it overrides IIS's calculation. Each pool thread watches for a network request and processes it, either by sending back a static file or by passing the request to an ISAPI extension DLL (such as ASP) or to a CGI. If the ISAPI extension processes a request synchronously and it takes a long time to process requests, then it will tie up the worker thread, leaving IIS with fewer worker threads to process other requests. For this reason, well-written ISAPI extensions, such as ASP, implement their own thread pools, place requests in a queue, and process them asynchronously with their own threads, so as not to tie up IIS worker threads. In general, if you find that the default limit of 256 threads is inadequate, you probably have a poorly written ISAPI extension tying up IIS worker threads. 256 is a lot of threads to have active simultaneously and will incur significant overhead in synchronization and context switching.

PoolThreadLimit is a hard limit that includes all IIS worker threads, including the HTTP, FTP, NNTP, and SMTP services. PoolThreadLimit will always be greater than or equal to MaxPoolThreads.

The ASP thread pool is a separate set of threads. Its size is controlled by the AspProcessorThreadMax metabase setting. The largest possible number of outstanding ASP requests is the sum of AspRequestQueueMax and AspProcessorThreadMax.

MaxPoolThreads REG_DWORD

Range: 0 - Unlimited

Default: 4 per processor

This parameter specifies the number of I/O worker threads to create per processor. Each pool thread watches for a network request and processes it. The MaxPoolThreads count does not include threads that are consumed by ISAPI applications; it refers only to the number of worker threads available to process request for static files. IIS will create more threads as needed to process ISAPI requests. The total number of IIS worker threads is capped by PoolThreadLimit.

By default, only four CGI applications can run concurrently. If you run many CGI applications, you should increase this value in order to increase the throughput. You could set the UsePoolThreadForCGI value (under ..\Services\W3SVC\Parameters) to FALSE (0); however, this is somewhat dangerous because it can significantly decrease performance during high usage of CGI applications. Generally, it is not good to create more than 20 threads per processor.

ListenBackLog REG_DWORD

Range: 1 to 300

Default: 15

This parameter specifies the maximum number of active connections to hold in the queue while they wait for server attention. Enhanced IIS 5.0 functionality generally makes it unnecessary to use or modify this entry, although with extremely heavy use it could be beneficial to increase this value up to 300.

Registry Path:
HKEY_LOCAL_MACHINE \SYSTEM
\CurrentControlSet
\Control
\SecurityProviders
\SCHANNEL
ServerCacheTime REG_DWORD

Range: 0 – Unlimited (measured in milliseconds)

Default: 300,000 milliseconds (5 minutes)

This parameter determines the amount of time an SSL session lasts. Once an SSL session has been established, clients can reconnect to that session at a fraction of the resource cost of the initial connection. If the SSL session expires, a new SSL session must be completely established. This parameter is not present by default. To change its behavior, you must add it to the registry. You should evaluate how long you expect SSL sessions to last, then set this parameter slightly longer. Do not make the timeout much longer than your estimate or this cache may begin to store stale data. For further discussion, see the Security section of this document.

Appendix 2: Tips for Optimizing Windows 2000 Web Server Performance

• Prior to upgrading to Windows 2000, you must uninstall Inoculan, PCAnywhere, and Veritas. You can install them again after you install Windows 2000.
• Run most of your applications in the new default medium protection mode (out-of-process pool). When applications are pooled, they share the same process, thereby reducing memory overhead. Also, running under medium protection allows for greater reliability than running your applications with low protection (in process).
• Check Event Logs for a high number of service restarts on both local and remote servers. If applications are failing frequently, performance will be very poor, but you might not notice the failures because reliable restarts are done automatically by the IISReset utility.
• Perform disk defragmentation from time to time on your servers. The files and directories on your server become fragmented over time. When this occurs, it takes Windows longer to gain access to files and directories because several additional disk reads are required to collect the various pieces. For information about the Windows 2000 Disk Defragmenter, see the Windows 2000 online documentation.
• If you are using SSL, make sure that the License Logging Service is enabled, even if anonymous users are accessing your Web servers.
• Do not routinely or periodically reboot IIS servers. Use the IISReset.exe utility instead. Servers should only be rebooted as a last resort. Also, any blue screens should be reported to PSS and solved, not ignored.

  Upgrade from IIS 4.0 to IIS 5.0 using one of these methods:

  • The rolling upgrade: upgrade one of your test servers before your entire suite of servers, and then upgrade your other computers.
  • The methodical upgrade: first create a set of tests for IIS 4.0, then run the tests on your computers prior to upgrading to IIS 5.0. Upgrade to IIS 5.0 and then run the tests again and measure performance differences between your old system and your new system.
• When possible, use unattended install scripts to simplify upgrades.
• If you are using Visual Basic objects, you will not see performance gains on apartment-threaded applications or on synchronous calls with global scope.
• High performance using Index Server 3.0 with Windows 2000 and IIS 5.0 requires some changes to the registry. For more information, see the Windows 2000 online documentation.

  If you used disk mirroring on Windows NT 4.0, be sure to do the following:

  • Make a good backup before the upgrade and leave the Windows NT 4.0 mirroring intact while upgrading to Windows 2000. This allows you to retain a legacy disk mirror.
  • Make sure that your mirrored disk drives are dynamic. Windows 2000 requires this. Before a drive can be converted to dynamic, you must have approximately 1MB of free space at the end of the disk. It may also be necessary to defragment the disk. For more information, see Knowledge Base article Q197738 - Not Enough Space Available to Upgrade to a Dynamic Disk.
  • In order to establish new mirrors, resize the partition before upgrading to Windows 2000.
  • See these additional Knowledge Base articles: Q175761 - Dynamic vs. Basic Storage in Windows 2000 and Q231376 - Legacy FT Sets Regenerate During a Windows 2000 Upgrade.
• Read the documents listed in the Resources section of this document.

Appendix 3: ASP Caching

The ASP Template Cache stores templates: pre-compiled ASP pages in text format (#includes have been evaluated, and so on). Its size is governed by the AspScriptFileCacheSize setting in the metabase, which defaults to 250. The ASP Script Engine Cache holds ASP templates that have been compiled into byte code. Its size is governed by the AspScriptEngineCacheMax setting in the metabase, which defaults to 125. The relationship between the two is that an ASP page is cached once in the template cache, but it can appear many times in the script engine cache if it is being executed on many threads simultaneously. A site with a lot of memory and a lot of distinct ASP pages that get hit often will probably want to increase AspScriptFileCacheSize (monitor ASP counters with System Monitor to diagnose). There is much less need to increase AspScriptEngineCacheMax; the main reason would be that the defaults are inadequate for machines with 8 or more processors. AspScriptEngineCacheMax should have a metabase value that is equal to or greater than the number of CPUs plus one, multiplied by AspProcessorThreadMax. AspProcessorThreadMax defaults to 25.

Every process that hosts ASP will have its own ASP Template and Script Engine Caches. By default that is just one process because ASP applications run at medium isolation in the pooled Dllhost process.

When ASP gets a request for a page, it checks the ASP Template Cache first. If there's an instance of that page cached there, the request is forwarded to the Script Engine Cache. If the requested page is not in the Template Cache, it is compiled into a template and forwarded to the ASP Script Engine Cache. If an instance of the page is cached in the Script Engine Cache and it is ready to run, that engine is run. Failing that, if there is an instance of the page that is already executing, ASP clones that running engine and executes the clone. This avoids the cost of re-parsing the template into byte code. If there is no script engine associated with the page, ASP takes the precompiled template from the ASP Template Cache, creates a new script engine and has it compile the template into byte code, and then executes it. When a page finishes executing, the script engine is put at the head of the free list. If the free list has grown larger than AspScriptEngineCacheMax, the least-recently used script engine is destroyed. A hit in the script engine cache means that ASP can avoid recompiling the template into byte code.

For more information about the metabase settings mentioned in this discussion, see Appendix 2: Performance Settings.

Appendix 4: Tips for getting the most out of an 8-processor machine.

IIS scales well up to four processors out of the box, but you will have to pay closer attention to tuning if you want IIS to scale up to an 8-processor machine. The applications that can best take advantage of more than four processors are the ones that involve heavy ASP processing and have heavy backend SQL dependencies. When testing the Web application, keep a close eye on the System: Context Switch/Sec, Processor: %Processor Time, Active Server Pages: Request Execution Time, Active Server Pages: Request/Sec and Active Server Pages: Request Queued counters. If the server has excessive context switching (tens of thousands/sec) and the CPU utilization is high (>80%) then you should look into decreasing the AspProcessorThreadMax setting.

As an example: On a particular application where the optimal AspProcessorThreadMax setting was 20 on a 4-processor machine, tests showed that the same setting to optimize an 8-processor machine was 5. At this setting, the application scaled 100% when going from 4 to 8 processors.

When lowering the AspProcessorThreadMax, you should see that the Active Server Pages: Request Execution Time and System: Context Switches/Sec counters both decrease, which will in turn allow IIS to use the process time more effectively and execute more ASP requests per second. When testing this, you should make sure that there is a small and bounded ASP Queue. This ensures that all the ASP threads are actively processing ASP requests, and you will get a better feel of what the maximum capacity is for the server.

An estimate of acceptable maximum queue size can be derived by dividing the response time that we are willing to accept, e.g., 3 seconds, by the measured execution time from Active Server Pages: Request Execution Time, e.g. 200 milliseconds. The portion of the response time contributed by the server to a single request is given roughly by the queue size, Active Server Pages: Request Queued, divided by the number of ASP worker threads (AspProcessorThreadMax * #processors) times the execution time for a single request, as the requests in the queue are ahead of the request that just arrived. (The execution time is fairly constant independent of queue size if context switches/sec is not exceedingly large.) So for the example above, a queue of no more than 600 requests should be acceptable, assuming we have 40 ASP worker threads and that we are willing to accept an added delay of 3 seconds to the response time.

When scaling up, it is also increasingly important to make sure that your machine is caching optimally (see Appendix 3). Given the increased performance of an 8-processor computer, you are more likely to get a bottleneck in disk performance. Instead of investing in a higher end disk sub system and more spindles, you can invest in more memory and increase the AspScriptFileCacheSize and AspScriptEngineCacheMax. IIS 5.0 can support up to 3GB of memory, so if your content is too large for IIS to cache fully, you will still have to build out your disk subsystem.

Appendix 5: Resources

General Information

• Killelea, Patrick. Web Performance Tuning. Cambridge, Massachusetts: O'Reilly & Associates, 1998. Contents also available at http://www.patrick.net/.
• Microsoft Corporation (Editors). Microsoft Windows 2000 Server Resource Kit, Microsoft Internet Information Services 5.0 Resource Guide. Redmond, Washington: Microsoft Press, 2000. In particular see the "Capacity Planning" and "Monitoring and Tuning Your Server" chapters.
• A Blueprint for Building Web Sites Using the Microsoft Windows DNA Platform at http://msdn.microsoft.com/msdn-online/start/features/dnablueprint.asp.
• The Microsoft Site Server 3.0 Capacity and Performance Resource List has a number of articles at http://www.microsoft.com/siteserver/ssrk/rk_list_all_capperf.htm.
• Capacity Model for Internet Transactions at http://www.microsoft.com/siteserver/ssrk/docs/rk_TCA.doc.
• The Microsoft TechNet site for IIS at http://www.microsoft.com/technet/iis/default.asp.
• Maximizing IIS Performance at http://www.microsoft.com/technet/iis/perflink.asp

Hardware Tuning

• Measuring Hardware Performance of Web Sites by Kathy Ferguson at http://www.microsoft.com/technet/iis/meashd.asp.
• Counters Quick Guide by Kathy Ferguson at http://www.microsoft.com/technet/iis/qguide.asp.

Tools

• The Web Application Stress tool and a tutorial for it are available at http://webtool.rte.microsoft.com/.
• Most of the tools discussed in the document are either available on the Windows 2000 Resource Kit companion CD or are built into the operating system. Please see the Windows 2000 online documentation for information on the latter.

Windows 2000 and IIS Performance and Tuning

• How To Set Up A Reliable Web Server Using Internet Information Server 5.0 and Windows 2000 at http://www.microsoft.com/technet/events/fall/tnq20005/html/default.htm.
• Monitoring and Optimizing Internet Information Server at http://www.microsoft.com/technet/iis/c06.asp.
• Tuning Internet Information Server Performance by Mike Moore at http://www.microsoft.com/isn/whitepapers/tuningiis.asp. Written for IIS 4.0.
• Windows 2000 Performance and Benchmarking (under the Reports and White Papers heading at http://www.microsoft.com/windows2000/guide/platform/performance/default.asp.
• Navigating the Maze of Settings for Web Server Performance Optimization, by Todd C. Wanke at http://www.microsoft.com/backstage/whitepaper.htm. Written for IIS 4.0.
• IIS 4.0 Tuning Parameters for High-Volume Sites by Michael Stephenson at http://msdn.microsoft.com/workshop/server/feature/tune.asp.

Testing and Tuning Web Applications

• 25+ ASP Tips to Improve Performance and Style by Len Cardinal and George Reilly at http://msdn.microsoft.com/workshop/server/asp/asptips.asp.
• Homer, Alex, et al. Professional Active Server Pages 3.0. London: Wrox Press, 1999. In particular, the chapter on optimizing ASP performance.
• Server Performance and Scalability Killers by George Reilly at http://msdn.microsoft.com/workshop/server/iis/tencom.asp.
• ASP Best Practices by George Reilly at http://support.microsoft.com/support/activeserver/AspBestPractices.ppt.
• Testing the Performance of Your Web Application by Matt Odhner at http://www.microsoft.com/technet/iis/wastip.asp.
• Improve the Performance of Your MDAC Application by Suresh Kannan at http://www.microsoft.com/data/impperf.htm

Security Issues

• Secure Internet Information Services 5.0 Checklist at http://www.microsoft.com/security/.
• Security Considerations for Network Attacks at http://www.microsoft.com/technet/security/dosrv.asp.

© 2000 Microsoft Corporation. All rights reserved.

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This white paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT.

Microsoft, Windows, and Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

Other product and company names mentioned herein may be the trademarks of their respective owners.

Microsoft Corporation • One Microsoft Way • Redmond, WA 98052-6399 • USA

¹ The default setting of 30 seconds for ObjectCacheTTL is too small to allow the IIS file cache to grow this large. See Appendix 1: Performance Settings.

² Note all medium- and high-isolation ISAPI and ASP applications run in instances of dllhost.exe. However, other COM+ server applications also run in their own instances of dllhost, so some dllhosts may have nothing to do with IIS. [What's the recommended way to figure out which they are?]