Windows Management and Scripting

A wealth of tutorials Windows Operating Systems SQL Server and Azure

Posts Tagged ‘high availability’

Powershell CMDLET for managing Windows Server 2012 Cluster

Posted by Alin D on January 17, 2013

If you manage Windows Failover Clusters, you may notice the Cluster.exe CLI command is missing after you install the Windows Server 2012 Failover Clustering feature. For years, systems administrators have used Cluster.exe to script the creation of clusters, move failover groups, modify resource properties and troubleshoot cluster outages. Yes, the Cluster.exe command still exists in the Remote Server Administration Tools (RSAT), but it’s not installed by default and is considered a thing of the past.

Another thing you may soon notice in Windows Server 2012 is the PowerShell and Server Manager Icons pinned to your taskbar. What you may not notice is that the default installation of the Windows Server 2012 operating system is now Server Core and contains more than 2,300 PowerShell cmdlets. Microsoft is sending a clear message that Windows servers should be managed just like any other data center server, both remotely and through the use of scripting. With Windows, that means PowerShell.

Fortunately, Windows Server Failover Clustering is no stranger to PowerShell. With Windows Server 2008 R2, 69 cluster-related PowerShell cmdlets assist with configuring clusters, groups and resources. This tip explores the new PowerShell cmdlets in Windows Server 2012 failover clusters.

With Windows Server 2012, a total of 81 failover cluster cmdlets can be used to manage components from PowerShell. New cluster cmdlets can perform cluster registry checkpoints for resources (Add-ClusterCheckpoint), monitor virtual machines for events or service failure (Add-ClusterVMMonitoredItem) and configure two new roles: Scale-Out File Servers (Add-ClusterScaleOutFileServerRole) and iSCSI Target Server (Add-ClusteriSCSITargetServerRole).
Windows PowerShell ISE

Windows PowerShell ISE

To list all the failover cluster cmdlets, use the PowerShell cmdlet “Get-command –module FailoverClusters” (Figure 1). I am using the built-in Windows PowerShell Integrated Scripting Environment (ISE) editor, which helps admins get familiar with all the failover clustering cmdlets.

In addition to the FailoverCluster cmdlets, Microsoft has several new modules of PowerShell cmdlets, including ClusterAwareUpdating with 17 new cmdlets, ClusterAware ScheduledTasks with 19 new cmdlets and iSCSITarget with 23 new cmdlets. There are many Cluster Aware Updating cmdlets, such as adding the CAU role (Add-CauClusterRole), getting an update report (Get-CauReport) or invoking a run to scan and install any new updates (Invoke-CauRun).

Cluster-Aware scheduled tasks are new to Windows Server 2012 and the Task Scheduler now integrates with failover clusters. A scheduled task can run in one of three ways:

ClusterWide on all cluster nodes
AnyNode on a random node in the cluster
ResourceSpecific on a node that owns a specific cluster resource

The new ScheduledTasks cmdlets create a cluster-aware scheduled task. In the table, you can see the cmdlets that register, get and set Clustered Scheduled task properties.
PowerShell Cmdlet     Description
Register-ClusteredScheduledTask     Creates a new clustered scheduled task
Unregister-ClusteredScheduledTask     Deletes a clustered scheduled task
Set-ClusteredScheduledTask     Updates existing cluster task
Get-ClusteredScheduleTask     Enumerates existing clustered tasks

To get an idea of how to use these PowerShell cmdlets, you first assign an action and trigger variable. The action variable specifies the program that is to be executed, such as the Windows calculator in the example below. The trigger variable sets up when the task is to be executed. The resulting cmdlets to schedule the task to run cluster-wide daily at 14:00 would look like this:

PS C:\> $action = New-ScheduledTaskAction –Execute C:\Windows\System32\Calc.exe

PS C:\> $trigger = New-ScheduledTaskTrigger -At 14:00 –Daily

PS C:\> Register-ClusteredScheduledTask -Action $action -TaskName ClusterWideCalculator -Description “Runs Calculator cluster wide” -TaskType ClusterWide -Trigger $trigger

TaskName         TaskType
——–         ——–
ClusterWideCa… ClusterWide

PS C:\>
Windows PowerShell Task Scheduler

While only PowerShell can be used to register, get/set and unregister Cluster-Aware scheduled tasks, you can use the Task Scheduler in Computer Management to view the cluster jobs (Figure 2).
iSCSI Target cmdlets
Cmdlets Failover Clusters3

Finally, failover clusters can now be configured with a highly available iSCSI Target Server. This role allows you to create and serve iSCSI LUNs in a highly available fashion to clients across your enterprise. To add this new cluster role, use the Cmdlet Install-WindowsFeature –name FS-iSCSITarget-Server (or use Server Manager) to install the iSCSI Target Server role. Then, use the new cmdlet Add-ClusteriSCSITargetServerRole to create the iSCSI Target resource and associate it with shared storage. You can then leverage the new iSCSI Target cmdlets to configure iSCSI LUNs (Figure 3).

There is no shortage of PowerShell cmdlets in Windows Server 2012 to help you manage your failover clusters. In addition to creating, configuring and troubleshooting your cluster, you can use PowerShell cmdlets to add new scale-out file server, iSCSI Target Server roles, clustered scheduled tasks and Cluster-Aware Updating.

Posted in Windows 2012 | Tagged: , , | Leave a Comment »

High availabitity more easy with Built-in NIC teaming in Windows Server 2012

Posted by Alin D on January 7, 2013

Thinking back a couple of years ago, I remember how painful and expensive the high availability options were for Windows and competing operating systems. Many Windows admins still experience the pain and cost of high availability in their environments, but Microsoft aims to fix this with NIC teaming in Windows Server 2012.

Be it for cloud scenarios or simple in-house setups, Windows Server 2012’s NIC teaming has a lot to offer in such a small package. It’s built right in and extremely simple to configure.

NIC teaming, or load balancing and failover, allows multiple NICs to be teamed together for bandwidth aggregation and failover in the event of a network hardware letdown. Until Windows Server 2012, we were at the mercy of NIC vendors to provide these features. There was no direct OS integration and Microsoft didn’t officially support NIC teaming. In Windows Server 2012, NIC teaming is front and center. It’s built right into the OS.

Some out-of-the-box NIC teaming features include:

  • Support for virtual NICs inside Hyper-V
  • Switch-dependent and switch-independent modes that do or do not require each NIC to be connected to the same Ethernet switch
  • VLAN traffic division so that applications can communicate with different VLANs at once
  • Support for up to 32 NICs per team

The only technologies that do not support NIC teaming in Windows Server 2012 are PCI I/O Virtualization, remote direct memory access and TCP Chimney, which are older technologies.

Configuring NIC teaming is a simple process that involves enabling it, adding a team on the server and configuring the specific network cards you wish to use for each team.

You can do this via PowerShell, the Server Manager GUI and via the RSAT tools in Windows 8. For PowerShell, you have a number of NIC teaming-specific commands such as:

  • NetLbfoTeam (Get, New, Remove, Rename, Set)
  • NetLbfoTeamMember (Add, Get, Remove, Set)
  • NetlbfoTeamNic (Get, New , Remove, Set)

Simply entering Get-Command -Module NetLbfo will show you all the commands available.

In the GUI, configuring a NIC team is a matter of:

  1. Selecting Local Server/Properties/Remote Desktop/NIC Teaming Administration.
  2. Selecting the server name and under Teams/Tasks, selecting New Team.
  3. Entering your team name and selecting the specific network adapters you want to use for the team.

More details about NIC teaming can be found in Microsoft’s document Windows Server 2012 NIC Teaming (LBFO) Deployment and Management.

One thing I’ve witnessed over the years is network admins not taking advantage of IT management and security controls they already have at their disposal. Having been in network administrator shoes, I think this is due in large part to a general lack of time and focus.  NIC teaming is not all that sexy, but it can buy you a ton of server resiliency. Customers, internal auditors and even management will never know you’re using it, but that’s what IT is about anyway: making things look simple by keeping the shop running smoothly. Microsoft is throwing us a bone with NIC teaming. I can’t think of any reason to not take advantage of it in Windows Server 2012.

Posted in TUTORIALS | Tagged: , , | Leave a Comment »

SCVMM PowerShell cmdlets can improve Hyper-V Live Migrations

Posted by Alin D on November 14, 2012

For small IT shops, PowerShell cmdlets improve on Hyper-V Live Migration functionality. But, with the addition of System Center Virtual Machine Manager and its PowerShell capabilities, those shops can precisely coordinate live migrations between nodes and Cluster Shared Volumes.

Microsoft System Center Virtual Machine Manager (SCVMM) includes a robust graphical user interface (GUI) from which administrators can perform Hyper-V live migrations. Even so, you may feel that SCVMM lacks the granular control needed to administer a complex, Hyper-V virtual infrastructure.

SCVMM PowerShell cmdlets, however, provide increased flexibility when managing Live Migration, far beyond what is possible with the GUI. With the following SCVMM PowerShell cmdlets and scripts, for example, you can live migrate an entire host’s worth of virtual machines (VMs) to a specific node, or transfer VMs based on their Cluster Shared Volumes assignments.

(Note: You must install the SCVMM console on the server or workstation from which you are running the script.)

Migrating all VMs from one node to another

Maintenance Mode is an SCVMM feature found within the graphical console that uses an Intelligent Placement algorithm to distribute all the VMs from the node of your choice to the remaining nodes in the cluster. But what if you want to maintain the same combination of VMs, just on a different node?

For example, in my virtual infrastructure, I have a mix of load balanced applications within a cluster. Placing load-balanced VM workloads on the same cluster node limits the effectiveness of the load balancing, especially in the event that a host fails.

Also, by maintaining the same mix of VMs on the destination node, you already know what the resource load will be. From my experience, even with a full free node in the cluster, SCVMM’s Maintenance Mode does not always move every source node VM to the empty destination node. That’s because the SCVMM Intelligent Placement feature constantly gathers load characteristics from each host to determine the best placement of VMs. As the free node receives additional VMs, its placement score goes down, causing SCVMM to distribute the remaining VMs to other cluster nodes.

From the GUI, there’s only one way to force the live migration of every VM on a node to another, named node: by manually going through the migration wizard for each VM.  But the following script overrides Intelligent Placement and synchronously move the VMs to a specific cluster node, simply by answering a few prompts.

# ——————————————————————————
# Migrate All VMs to Alternate Node in the Same Cluster
# ——————————————————————————

Add-PSSnapin Microsoft.SystemCenter.VirtualMachineManager

$VMM = Read-Host “Enter Name of VMM Server”
$SH = Read-Host “Enter Source Host Name”
$DH = Read-Host “Enter Destination Host Name”

Get-VMMServer -computername $VMM
Get-VM | ? {$_.hostname -like “$SH*”} | Move-VM -VMHost “$DH*”

To run the script, follow these steps:

  1. Save the SCVMM PowerShell script above (e.g., MigrateAllVMsOnNode_SCVMM.ps1)
  2. Open Windows PowerShell.
  3. Run the script.
  4. Answer the prompts for SCVMM server, source node name and a destination node within the same cluster.Migrate All VMs On Node
  5. Follow progress of the migration from the command status, Failover Cluster Manager or the SCVMM Jobs page.

Check Status on Console

Migrating VMs on a Cluster Shared Volume between nodes

This script identifies the VMs on a particular Cluster Shared Volume, and live migrates them to a specified destination node. Use this script to keep VMs on a Cluster Shared Volume together on the same host after a live migration.

Hyper-V Cluster

Why would you want to do this? During normal cluster operations, a VM resource can utilize direct I/O from a volume shared between all nodes, and any node in the cluster can own that VM resource. But problems arise if you use Microsoft Data Protection Manager or other backup software based on Hyper-V Volume Shadow Copy (VSS).  During a backup, only the node that owns the VM’s CSV has direct access to disk I/O. VMs that live on other nodes but share the same CSV will have their I/O redirected over the network, causing disk latency and degraded performance.

To avoid this issue, enact the following placement architecture to maintain full disk I/O with no deprecated performance for all VMs on that CSV volume. To reorganize VMs after using Maintenance Mode, use this script to easily live migrate just the VMs on a particular CSV to the desired source node.


# ——————————————————————————
# Live Migrate Virtual Machines On a Particular Volume to a New Host in Same Cluster
# ——————————————————————————

Add-PSSnapin Microsoft.SystemCenter.VirtualMachineManager

$VMM = Read-Host “Enter Name of VMM Server”
$SH = Read-Host “Enter Source Host Name”
$DH = Read-Host “Enter Destination Host Name”
$Vol = Read-Host “Enter Volume/CSV Volume to Move VMs to Destination Host”

Get-VMMServer -computername $VMM
Get-VM | ? {$_.hostname -like “$SH*”} | ? {$_.Location -like “*$Vol*”} | Move-VM -VMHost “$DH*”

  1. Save the PowerShell script above (e.g. MigrateAllVMsByVolume_SCVMM.ps1).
  2. Open Windows PowerShell.
  3. Run the script that you saved above.
  4. Answer the prompts for SCVMM server, source node name, destination node and the Cluster Shared Volume of VMs that you want to target.
  5. Follow progress of the migration from the command status, Failover Cluster Manager or the SCVMM Jobs page.

Further experimentations with SCVMM PowerShell cmdlets
The above scripts are just two examples of what you can do with the SCVMM PowerShell cmdlets. Because of the depth of information that SCVMM gathers about each VM, there are more ways to include or exclude VMs for Live Migration:

  • Name: You can live migrate VMs with a certain name attribute (e.g., using the <tt>–like</tt>command option).
  • Memory:  You can target VMs that are above or below a certain memory threshold.
  • Operating system:  You can select virtual machines by operating system, such as Windows, Linux (quick migration only), etc.


With SCVMM PowerShell cmdlets, you can customize your Live Migration experience far beyond what’s possible in the graphical console. System Center Virtual Machine Manager 2012 will add options within its updated cmdlet, but the GUI will still lack granular management functionality. As such,  it is in your best interest to learn how to streamline administrative tasks with SCVMM PowerShell cmdlets.

Posted in System Center | Tagged: , , | Leave a Comment »

Best practices for SQL Clustering

Posted by Alin D on June 8, 2011

SQL Server clustering is a high-availability technology for SQL Server instances. It involves the sharing of server resources between one or more nodes (or servers), which have one or more shared disks grouped into logical units called resource groups. A resource group containing at least one IP address, network name and disk resource is called a virtual server. The cluster service arbitrates ownership of the resource groups. A single node can own a resource group and its associated resources at any given time.

Clustering basics

Each virtual server appears on the network as a complete system. When the virtual server contains SQL Server resources, clients connected to the virtual server access resources on its current host node. While the terms “active” and “passive” are often used in this context, they are not fixed roles, as all nodes in a cluster are interchangeable. Should the current host, sometimes designated as the primary, fail, the resource group will be transferred to another node (secondary node) in the cluster. With clusters having more than two nodes or two instances, it is important to set failover order by choosing the preferred node ownership order for each instance. The secondary will become the primary and host the virtual server. Active client connections will be broken during failover, but they can reconnect to the virtual server now hosted by the new node. The clients will have to reconnect manually, and work in progress will be lost during the failover. Most commercial applications now handle this reconnection task seamlessly.

The goal of clustering is to provide increased availability to clients by having a hot standby system with an automatic failover mechanism. SQL Server clustering is not a load-sharing or scale-out technology. On all clusters during a failure there will be a brief database server interruption. On large clusters with multiple nodes and instances, clients may experience degraded performance during a failure event but they will not lose database availability.

Clustering topologies

There are four types of cluster topologies — or arrangements of nodes in a cluster:

  • Single instance
  • Multi-instance
  • N+1
  • N+M

Single instance:

In this case, one node in a cluster owns all resource groups at any one time and the other nodes are offline. Should the primary node owning the resources fail, the resource groups will be transferred to the secondary node, which comes online. While the secondary node comes online, it will assume ownership of the resource groups, which typically consist of disks containing your database files and transaction logs. The secondary node comes online (starts up), and SQL Server will start up on the virtual server and roll uncommitted transactions in the transaction log backward or forward as it recovers the database.

This topology was formerly called active-passive. Single-instance clustering is most frequently used for mission-critical applications, where the cost of downtime far outweighs the cost of the wasted hardware resources of the secondary node sitting idle while offline.

Multiple instance:

In this situation, one virtual server in a cluster owns some of the resource groups and another virtual server owns other resource groups. At any one time, the virtual servers themselves can be hosted by a single node or different nodes and would appear to clients as named instances of a single server. In that case, they are named instances of a virtual server, hence the name multiple instance. With multiple-instance clustering, previously called active-active, the hardware requirements of each individual node are greater as each node may at any one time be hosting two (or more) virtual servers.

You should consider multiple-instance clusters to be more cost effective than single-instance clusters as there are no nodes offline or waiting. However, should one node host more than one virtual server, performance for clients is typically degraded. Your best bet is to use multiple instances when you require high availability but not high performance.


This is a modification of multiple-instance clustering topologies where two or more nodes share the same failover node. The secondary node will need enough hardware capabilities to support the load of all N servers at any one time should they all fail over simultaneously. You can achieve cost savings if multiple clusters use the same failover node. However, the cost of an individual

node tends to be small in comparison to other related clustering costs, such as storage.

Many people consider N+1 to be more cost effective than multiple-instance clustering because there is only one secondary node offline (or waiting) for several active nodes. However, depending on the hardware configuration of the failover node, it does not offer the performance of multiple-instance clustering. Use N+1 in environments where cost constraints force you to reduce the number of failover nodes and you need high availability but not high performance.


In a situation where you have two or more working nodes in a cluster along with two or more standby nodes, it is typically configured in eight-node clusters with six working nodes for every two standby, or five working nodes for every three standby.

N+M offers some of the cost benefits of N+1, but it has a lower chance of performance degradation during a multiple failure event than N+1 since the failover node(s) do not have to support the entire load of the failed nodes. Use N+M in environments where cost constraints force you to reduce the number of failover nodes and at the same time provide a high level of performance.

Clustering dependencies

SQL Server clustering has several dependencies:

  • Network
  • Hardware
  • Software

Network dependencies:

Clustering requires a private network among all nodes in a cluster. Clustering services use a private communication channel on each node to keep in sync with each other. This allows the cluster to communicate and act appropriately even if the public network is offline. Looks-Alive and Is-Alive checks — used by cluster services to determine if a cluster resource group is “up” — connect over the public networks to best emulate a client connection process.

Hardware dependencies:

Clustering requires specialized hardware and software. And to share resources between nodes, you need specialized disk controllers. Clustering hardware must be certified by Microsoft to meet the requirements of clustering. And, you must have a second set of network cards to provide the private network between cluster nodes.

Software dependencies:

To benefit from clustering services, you need specialized versions of the operating system (Windows 2000 and 2003 Enterprise or Data Center editions). You will also need SQL Server 2000 Enterprise Edition, SQL Server 2005 Standard Edition (up to two nodes) or SQL Server 2005 Enterprise Edition (up to eight nodes).

Clustering best practices

What follows is a list of clustering best practices. I have broken these down according to dependencies.

Network best practices

There are two different and contradictory settings required for the public network and the private network in clustering.


Ensure the private network is private. Clustering requires a 150-ms ping response time. If your private network is saturated or congested with other network traffic, you may find your clusters failing over unexpectedly. On your private network, consider isolating traffic by implementing a VLAN (virtual LAN), a separate subnet or use a crossover cable for Single-instance clusters. The actual traffic generated by cluster communication is small, so high-bandwidth networks are unnecessary. However, they must still be low latency and reliable. Make sure the following points are established on the private network:

  • Use TCP/IP as the only protocol bound to the NIC.
  • No default gateway is configured.
  • No DNS servers are configured unless the cluster nodes are DNS servers, in which case should be configured.
  • No DNS registration or DNS suffix is configured.
  • No WINS servers are configured.
  • Static IP addresses are used for all nodes.
  • NetBIOS over TCP/IP is disabled.
  • No NIC teaming is used, where two network interface cards are aggregated together to act as a single NIC card.


For your public network, use at least two WINS or DNS servers on your cluster network segment or VLAN. While installing your cluster you will have to resolve cluster, DC (domain controller) and virtual server names. You must have a name server on your network for this. You can decrease the time required for a node to fail over by providing a name server on your network as well.

Use at least two DCs on your network. Clustering requires DCs not only during setup but also for normal functioning and failover.

If you use NIC teaming for greater bandwidth throughput and reliability, do not configure it while building the cluster. Add NIC teaming as a last step before final testing. Be prepared to “undo” NIC teaming as an early step in troubleshooting. Microsoft Customer Support Services (CSS) will likely direct you to disable teaming as a first diagnostic step, so be ready.


Ensure that your network card settings are identical for every server in your cluster and that they are not configured to automatically detect network settings.

Software best practices

Ensure applications are cluster aware and will not lose work or fail to meet the SLA during a cluster failover.

Ensure transactions are as small as possible in your application and on any jobs that may run on your clustered SQL Servers. Long-running transactions increase the length of time required to apply the transaction log on the failover node and consequently increase the amount of time for failover.

Do not run antivirus software on cluster nodes. If you must run antivirus software, be sure the quorum disk and database files are excluded from the scans. Even in this configuration, there have been reports of antivirus drivers interfering with cluster disk resource failover. Test your setup and make sure it fails as expected. Select another antivirus product if yours causes problems.

Make sure there are no password expiration policies in use for any of the cluster-related accounts. Cluster accounts should:

  • be the same for all nodes in the cluster;
  • have domain accounts (but not domain admin accounts) and have local administrative rights on each node in the cluster. SQL Server 2005 forces you to set up domain-level groups for these accounts and then grants appropriate rights to the groups.
  • have the least security privileges to minimize damage that could be done to the node or other servers on your network should the password be compromised or the account be hijacked by a buffer overflow.

Ensure all software components are the same version (i.e., SQL Server 2005 Standard), same architecture (i.e., 64 bit for all OS and SQL Server components) and at the same service pack and hot fix level. The exception is that individual SQL Server instances can be at different releases, editions and hotfix levels.

Ensure all external software dependencies (COM components, file paths, binaries) are either cluster aware or installed on all nodes in a cluster. MSDTC (Microsoft Distributed Transaction Coordinator) is the most common external dependency in a cluster. While it is not necessary, many people install it before installing SQL Server because installing it later is much harder.

When installing a cluster, consider installing a single-node cluster and adding nodes to the cluster as required. This way, if the cluster setup fails while adding a single node, you are left with a working cluster (although it could be a single-node cluster).

While applying hot fixes or service packs that require a system reboot, apply it to the primary (current instance host), fail over to the secondary, reboot the primary, fail back to the primary and reboot the secondary. Typically hot fixes and service packs are cluster aware and install on all cluster nodes simultaneously.


Ensure that your cluster is approved by the vendor and that it is part of the Microsoft Windows Catalog with a specific endorsement for clustering.

Ensure each node in your cluster has identical hardware and components.

Regularly check vendor Web sites for potential hardware problems, fixes and BIOS patches for each component in your cluster.

Use the appropriate RAID technology to ensure that your disk array is fault tolerant. Be as proactive as possible in replacing failed or marginal disks. A disk failure will put a greater load on the remaining disks in an array and may cause other marginal disks to fail. Depending on your RAID technology, your RAID array may not be tolerant to more than one disk failure per array.

Ensure you have properly conditioned or charged batteries on any array controlle. It prevents data loss or corruption in the event of a power failure.

Use uninterrupted power supplies and be sure you have redundancy in your power supplies.

Use Hot-Add Memory if it’s supported by your SQL Server version, operating system and hardware. Hot-Add Memory is a hardware technology that allows you to add memory to a running system; the OS detects and uses the additional memory. Windows Server 2003, Enterprise and Data Center Editions, as well as SQL Server 2005 Enterprise Edition can take advantage of Hot-Add Memory. Read about Hot-Add Memory Support in Windows Server 2003.

Use ECC (Error Correction Code) memory chips, which store parity information used to reconstruct original data when errors are detected in data held in memory.

Use fault-tolerant NICs and network devices (switches).


Clustering is a relatively new technology and has a reputation for being fragile. SQL Server 2000 clustering is far simpler than the earlier versions and has proven to be much more reliable. Today, clustering on SQL Server 2000 and SQL Server 2005 is a highly reliable technology, but it still has many dependencies that prevent it from meeting your high-availability goals. Foremost among these dependencies is a staff that is trained and knowledgeable. Running a close second is having operating processes and procedures that are designed to work specifically with a SQL Server cluster. Ensure that you address all of your clustering dependencies to deliver high availability with SQL Server clustering.

Posted in SQL | Tagged: , , , , , , , , , , , , , , , , , , | Leave a Comment »


Get every new post delivered to your Inbox.

Join 682 other followers

%d bloggers like this: