Managing more than a handful of Dell servers becomes cumbersome using the iDRAC web interface.  This is especially painful with several hundred servers or more.  Luckily Dell provides CLI tools you can use with XML templates to automate provisioning and administration for you.

Why use XML Templates?
We tie in a bunch of other Open Source tools like Foreman, Ansible and Puppet to provide a seamless lifecycle and management workflow.  Dell has proprietary software to manage iDRACs but using the innate XML template functionality within the iDRAC better fits our provisioning workflow with CLI tools and orchestration.  Lastly, being able to revision control things into an SCM like git is important for us.

Racadm and Dell Automation
You can also just do a lot using ssh and racadm commands and we’ll cover the most common examples that are useful to us.  You should take a look at other racadm guides for a more exhaustive example of racadm administration.

Requirements:
You’ll need a few basic things before you proceed:

• An EL5, EL6, EL7-based Linux host or VM
• Access to the internet and access to the iDRAC interface you want to manage
• A NFS server export (for storing/serving XML templates)

Download the iDRAC CLI Repo
Repo instructions are taken from Dells site here

wget -q -O - http://linux.dell.com/repo/hardware/latest/bootstrap.cgi | bash

Piping anything into bash is generally a bad idea, but this a vetted method.
This will setup a few extra yum-specific plugins needed, and a few dependencies.

Downloading GPG key: http://linux.dell.com/repo/hardware/latest/RPM-GPG-KEY-dell
    Importing key into RPM.
Downloading GPG key: http://linux.dell.com/repo/hardware/latest/RPM-GPG-KEY-libsmbios
    Importing key into RPM.
Write repository configuration
Downloading repository RPM
Installing repository rpm: http://linux.dell.com/repo/hardware/latest/platform_independent/rh60_64/prereq/dell-omsa-repository-2-5.noarch.rpm
Installing yum plugins for system id
Loaded plugins: product-id, security, subscription-manager, versionlock

Dependencies installed:

================================================================================
 Package                Arch      Version              Repository          Size
================================================================================
Installing:
 yum-dellsysid          x86_64    2.2.27-4.4.1.el7     dell-omsa-indep     16 k
Installing for dependencies:
 python-smbios          x86_64    2.2.27-4.4.1.el7     dell-omsa-indep     69 k
 smbios-utils-python    x86_64    2.2.27-4.4.1.el7     dell-omsa-indep     63 k

Transaction Summary
================================================================================
Install  1 Package (+2 Dependent packages)

Install iDRAC CLI tools
This will pull in smbios-utils-bin, srvadmin-argtable2, srvadmin-omilcore

yum install srvadmin-idracadm openssl-devel

How to Use the iDRAC CLI
There are a handful of very useful ways to use the iDRAC CLI, almost any action you’d normally have to do in a browser or remote OOB Java viewer you can do with the tool.  It can also parse and read XML templates to configure the iDRAC.

• 10.1.8.30:/srv/configs is our local NFS (RW) share in these examples.

Managing iDRAC XML Templates
iDRAC automation centers around manipulation of XML templates which comprise the configurations and settings of both the iDRAC and the attached peripherals (PERC cards, etc).

Changes will revolve around pulling, editing, pushing templates.
It’s also useful to revision control these if you like via git, but entirely optional.

Pulling iDRAC Template
This is useful for safekeeping before editing and also to modify and use for pushing a similar configuration across many many systems.

idracadm -r 10.1.8.59 -u root -p PASSWORD get -t xml -f host.xml -l 10.1.8.30:/srv/configs

Now you should have an XML file called host.xml or whatever you called it sitting in the /srv/configs NFS share.  It’s a good idea to backup all of your configs this way before doing any major modifications.

Pushing iDRAC Template
You can now edit the XML file to change or add attributes.  You can then push this XML file to your iDRAC for application by substituting get with set above.

idracadm -r 10.1.8.59 -u root -p PASSWORD set -t xml -f host-edit.xml -l 10.1.8.30:/srv/configs

Example:  Adding new local iDRAC Users
In this example you can easily add new users to any number of iDRACS.  Note the Users.4 below and be sure to increment this for future additions (or track it somewhere).  We simply appended Users.4 with a new user which will auto-create on the DRAC end.  Future users would use Users.5 and so forth, iterating upwards.

<!-- <SystemConfiguration> -->
<SystemConfiguration Model="PowerEdge R620" ServiceTag="3H5XXXX" TimeStamp="Wed Dec 11 11:38:15 2013">
<!--Exported configuration does not contain FC-HBA data as the presence of the appropriate cards could not be detected!-->
<Component FQDD="iDRAC.Embedded.1">
 <Attribute Name="IPMILan.1#Enable">Disabled</Attribute>
 <Attribute Name="IPMILan.1#PrivLimit">Administrator</Attribute>
 <Attribute Name="IPMILan.1#EncryptionKey">0000000000000000000000000000000000000000</Attribute>
 <Attribute Name="IPMILan.1#CommunityName">public</Attribute>
 <Attribute Name="IPMILan.1#AlertEnable">Disabled</Attribute>
 <Attribute Name="Users.4#UserName">qe-scale</Attribute>
 <Attribute Name="Users.4#Password">********</Attribute>
 <Attribute Name="Users.4#Privilege">507</Attribute>
 <Attribute Name="Users.4#IpmiLanPrivilege">Operator</Attribute>
 <Attribute Name="Users.4#IpmiSerialPrivilege">Operator</Attribute>
 <Attribute Name="Users.4#Enable">Enabled</Attribute>
 <Attribute Name="Users.4#SolEnable">Enabled</Attribute>
 <Attribute Name="Users.4#ProtocolEnable">Enabled</Attribute>
 <Attribute Name="Users.4#AuthenticationProtocol">SHA</Attribute>
 <Attribute Name="Users.4#PrivacyProtocol">AES</Attribute>
</Component>
</SystemConfiguration>

Example:  Carving out RAID Disks
You can also carve out RAID disks with the XML templates.  The easiest way to do this is to set one machine up manually, pull the configuration down and then push it to other iDRACs.  They should reboot into the PERC menu and do the right thing.

Example: Changing the Boot Order of NICs
Sometimes it’s useful to switch the order of what NICs will PXE first, you can do this with an XML template.  This is useful in environments where a lot of automated bare-metal provisioning might be needed.

• .1.1.1 = first NIC
• .1.2.1 = second NIC

You can view the current boot order from the iDRAC via racadm commands

ssh mgmt-file01.example.com racadm get BIOS.BiosBootSettings.BootSeq

Note: If you add new network interfaces you’ll need to reboot the iDRAC for them to show up in this listing so they can be manipulated.

You can modify this order in the XML below:

 <Attribute Name="BiosBootSeq">NIC.Integrated.1-2-1, HardDisk.List.1-1, NIC.Integrated.1-1-1</Attribute>

Viewing The current Job ID
It’s easy to make mistakes in XML, so you will want a way to view the status of a job.
You can view the status of any job by looking at it’s JID or Job ID.

idracadm -r 10.1.8.59 -u root -p PASSWORD jobqueue view -i JID_404441080579

---------------------------- JOB ------------------------- 
[Job ID=JID_404441080579]
Job Name=Export: System configuration XML file
Status=Completed
Start Time=[Not Applicable]
Expiration Time=[Not Applicable]
Message=[SYS043: Successfully exported system configuration XML file.]
----------------------------------------------------------

Viewing All Prior Jobs
You can simply list the status of all jobs run also.

idracadm -r 10.1.8.59 -u root -p PASSWORD jobqueue view

-------------------------JOB QUEUE------------------------ 
[Job ID=JID_365611035667]
Job Name=Configure: Import system configuration XML file
Status=Completed
Start Time=[Not Applicable]
Expiration Time=[Not Applicable]
Message=[SYS053: Successfully imported and applied system configuration XML file.]
----------------------------------------------------------
[Job ID=JID_254163911329]
Job Name=Export: System configuration XML file
Status=Completed
Start Time=[Not Applicable]
Expiration Time=[Not Applicable]
Message=[SYS043: Successfully exported system configuration XML file.]
----------------------------------------------------------

Other Useful iDRAC Commands
You can reboot a bare-metal server remotely (useful for automating large-scale systems provisioning) via the following command:

idracadm -r 10.1.8.59 -u root -p PASSWORD serveraction powercycle

Rebooting a remote iDRAC.

idracadm -r 10.1.8.59 -u root -p PASSWORD racreset soft

You can also reboot a local iDRAC directly from the CLI of the server after querying it for information.  This is useful in case of OOB failure/troubleshooting:

/opt/dell/srvadmin/sbin/srvadmin-services.sh start
/opt/dell/srvadmin/sbin/racadm getsysinfo 2>&1 | less
/opt/dell/srvadmin/sbin/racadm racreset

Determining Firmware Version
You can use this command to query the firmware version of the iDRAC.

ssh root@mgmt-host racadm getconfig -g idracinfo -o idRacVersionInfo 2>/dev/null

Updating the lifecycle controller firmware via racadm.
You can use an anon FTP server with the extracted payload/firmimg file.  This can be extracted from the .EXE as it’s really just a zip archive.  The example below is an iDRAC update with the 2.41.40.40 version renamed from payload/firmimg.d7

ssh mgmt-c10-h07-r730xd.example.com racadm fwupdate -f 10.12.64.163 ftp user -d pub/2.41.40.40.d7

Determining the BIOS version
Like the racadm firmware version, you can do the same to query the running BIOS version.

ssh root@mgmt-host racadm getsysinfo | grep -i bios | awk '{print $5}'

Updating the System BIOS via racadm
You can also use racadm for updating the BIOS as well, we use files hosted on a NFS share for this, you’ll first want to copy the .EXE file (it’s really just an archive) on an accessible NFS share that can be mounted by your iDRAC.

This command will reboot the system once the BIOS update is complete, if you don’t want this to happen remove the –reboot flag and do it yourself later.

ssh root@mgmt-host racadm update -f BIOS_NGYCX_WN64_2.7.0_01.EXE -l 10.1.39.29:/srv/configs/R620 --reboot

Caveat – iDRAC username Length
We found a bug where if you exceed the username character limit of 16 characters the iDRAC will hard lock, requiring a manual hands-on reset or worse.  We’ve filed this issue with Dell but aren’t sure if it’s fixed or not.  Avoid usernames greater than 16 characters.

Caveat – iDRAC IP vs. FQDN
In later versions of the iDRAC firmware there’s a bug in DNS lookups that fail when using a FQDN with racadm to push commands.  Use the IP address to work around this.  We’ve traced this down via GDB and strace it and seems to fail getaddr() (we’ve also notified Dell).

Fix Hung iDRAC Tasks
Occassionally the iDRAC may get hung up on job requests not completing and piling up, rebooting will not fix this.  The following command may work to unwedge them.

racadm -r drac.ip.address -u root -p password jobqueue delete -i JID_CLEARALL_FORCE

Then wait about 120 seconds and reboot the iDRAC again

racadm –r drac.ip.address –u root –p password racreset

Usage and Extension
We utilize the racadm CLI tools with a simple shell for loop to quickly provision multiple racks with only a few commands.  We tie this into Foreman for full lifecycle deployment and management and tools like Ansible and Puppet to provide configuration management.

Using Ansible to Manage Dell Servers via Racadm
You can do a lot with Ansible, here’s some Ansible playbooks we use to do things like toggle NIC interface boot order or also change RAID configuration remotely and other things.  Feel free to use/borrow these as examples and build upon them further.

Automating Everything
When deploying new servers you can tie all this together.  Everyones infrastructure is different but this is how we do it:

• Obtain MAC address information beforehand of any new servers and iDRAC
• Prep racadm commands with proper templating to fire off when servers are racked
• Prep your DHCPD entries for the iDRAC interfaces so they come online when powered on
• Prep the Foreman Hammer CLI commands to create host entries, set to build
• Once servers are powered on, run your deployment script
  • racadm commands to create accounts, settings, carve disks etc.
  • Foreman Hammer CLI commands to provision bare-metal hosts
• Optionally, automatically populate a Wiki based on the new servers
  • If you used TRAC wiki, this rack generation tool might be useful

This set of examples is by no means exhaustive, experiment and see what else you can do with it.  We’ve found almost everything you normally had to do within a browser or Java OOB can be done by referencing parameters in the iDRAC XML template.  I take no responsibility for any damage that may be done.  Please comment if you’ve got new use cases we can add.