Managing more than a handful of Dell servers becomes cumbersome using the iDRAC web interface. This is especially painful with several hundred servers or more. Luckily Dell provides CLI tools you can use with XML templates to automate provisioning and administration for you.
Why use XML Templates?
We tie in a bunch of other Open Source tools like Foreman, Ansible and Puppet to provide a seamless lifecycle and management workflow. Dell has proprietary software to manage iDRACs but using the innate XML template functionality within the iDRAC better fits our provisioning workflow with CLI tools and orchestration. Lastly, being able to revision control things into an SCM like git is important for us.
Racadm and Dell Automation
You can also just do a lot using ssh and racadm commands and we’ll cover the most common examples that are useful to us. You should take a look at other racadm guides for a more exhaustive example of racadm administration.
You’ll need a few basic things before you proceed:
- An EL5, EL6, EL7-based Linux host or VM
- Access to the internet and access to the iDRAC interface you want to manage
- A NFS server export (for storing/serving XML templates)
Download the iDRAC CLI Repo
Repo instructions are taken from Dells site here
wget -q -O - http://linux.dell.com/repo/hardware/latest/bootstrap.cgi | bash
Piping anything into bash is generally a bad idea, but this a vetted method.
This will setup a few extra yum-specific plugins needed, and a few dependencies.
Downloading GPG key: http://linux.dell.com/repo/hardware/latest/RPM-GPG-KEY-dell Importing key into RPM. Downloading GPG key: http://linux.dell.com/repo/hardware/latest/RPM-GPG-KEY-libsmbios Importing key into RPM. Write repository configuration Downloading repository RPM Installing repository rpm: http://linux.dell.com/repo/hardware/latest/platform_independent/rh60_64/prereq/dell-omsa-repository-2-5.noarch.rpm Installing yum plugins for system id Loaded plugins: product-id, security, subscription-manager, versionlock
================================================================================ Package Arch Version Repository Size ================================================================================ Installing: yum-dellsysid x86_64 2.2.27-4.4.1.el7 dell-omsa-indep 16 k Installing for dependencies: python-smbios x86_64 2.2.27-4.4.1.el7 dell-omsa-indep 69 k smbios-utils-python x86_64 2.2.27-4.4.1.el7 dell-omsa-indep 63 k Transaction Summary ================================================================================ Install 1 Package (+2 Dependent packages)
Install iDRAC CLI tools
This will pull in smbios-utils-bin, srvadmin-argtable2, srvadmin-omilcore
yum install srvadmin-idracadm openssl-devel
How to Use the iDRAC CLI
There are a handful of very useful ways to use the iDRAC CLI, almost any action you’d normally have to do in a browser or remote OOB Java viewer you can do with the tool. It can also parse and read XML templates to configure the iDRAC.
- 10.1.8.30:/srv/configs is our local NFS (RW) share in these examples.
Managing iDRAC XML Templates
iDRAC automation centers around manipulation of XML templates which comprise the configurations and settings of both the iDRAC and the attached peripherals (PERC cards, etc).
Changes will revolve around pulling, editing, pushing templates.
It’s also useful to revision control these if you like via git, but entirely optional.
Pulling iDRAC Template
This is useful for safekeeping before editing and also to modify and use for pushing a similar configuration across many many systems.
idracadm -r 10.1.8.59 -u root -p PASSWORD get -t xml -f host.xml -l 10.1.8.30:/srv/configs
Now you should have an XML file called host.xml or whatever you called it sitting in the /srv/configs NFS share. It’s a good idea to backup all of your configs this way before doing any major modifications.
Pushing iDRAC Template
You can now edit the XML file to change or add attributes. You can then push this XML file to your iDRAC for application by substituting get with set above.
idracadm -r 10.1.8.59 -u root -p PASSWORD set -t xml -f host-edit.xml -l 10.1.8.30:/srv/configs
Example: Adding new local iDRAC Users
In this example you can easily add new users to any number of iDRACS. Note the Users.4 below and be sure to increment this for future additions (or track it somewhere). We simply appended Users.4 with a new user which will auto-create on the DRAC end. Future users would use Users.5 and so forth, iterating upwards.
<!-- <SystemConfiguration> --> <SystemConfiguration Model="PowerEdge R620" ServiceTag="3H5XXXX" TimeStamp="Wed Dec 11 11:38:15 2013"> <!--Exported configuration does not contain FC-HBA data as the presence of the appropriate cards could not be detected!--> <Component FQDD="iDRAC.Embedded.1"> <Attribute Name="IPMILan.1#Enable">Disabled</Attribute> <Attribute Name="IPMILan.1#PrivLimit">Administrator</Attribute> <Attribute Name="IPMILan.1#EncryptionKey">0000000000000000000000000000000000000000</Attribute> <Attribute Name="IPMILan.1#CommunityName">public</Attribute> <Attribute Name="IPMILan.1#AlertEnable">Disabled</Attribute> <Attribute Name="Users.4#UserName">qe-scale</Attribute> <Attribute Name="Users.4#Password">********</Attribute> <Attribute Name="Users.4#Privilege">507</Attribute> <Attribute Name="Users.4#IpmiLanPrivilege">Operator</Attribute> <Attribute Name="Users.4#IpmiSerialPrivilege">Operator</Attribute> <Attribute Name="Users.4#Enable">Enabled</Attribute> <Attribute Name="Users.4#SolEnable">Enabled</Attribute> <Attribute Name="Users.4#ProtocolEnable">Enabled</Attribute> <Attribute Name="Users.4#AuthenticationProtocol">SHA</Attribute> <Attribute Name="Users.4#PrivacyProtocol">AES</Attribute> </Component> </SystemConfiguration>
Example: Carving out RAID Disks
You can also carve out RAID disks with the XML templates. The easiest way to do this is to set one machine up manually, pull the configuration down and then push it to other iDRACs. They should reboot into the PERC menu and do the right thing.
Example: Changing the Boot Order of NICs
Sometimes it’s useful to switch the order of what NICs will PXE first, you can do this with an XML template. This is useful in environments where a lot of automated bare-metal provisioning might be needed.
- .1.1.1 = first NIC
- .1.2.1 = second NIC
You can view the current boot order from the iDRAC via racadm commands
ssh mgmt-file01.example.com racadm get BIOS.BiosBootSettings.BootSeq
Note: If you add new network interfaces you’ll need to reboot the iDRAC for them to show up in this listing so they can be manipulated.
You can modify this order in the XML below:
<Attribute Name="BiosBootSeq">NIC.Integrated.1-2-1, HardDisk.List.1-1, NIC.Integrated.1-1-1</Attribute>
Viewing The current Job ID
It’s easy to make mistakes in XML, so you will want a way to view the status of a job.
You can view the status of any job by looking at it’s JID or Job ID.
idracadm -r 10.1.8.59 -u root -p PASSWORD jobqueue view -i JID_404441080579
---------------------------- JOB ------------------------- [Job ID=JID_404441080579] Job Name=Export: System configuration XML file Status=Completed Start Time=[Not Applicable] Expiration Time=[Not Applicable] Message=[SYS043: Successfully exported system configuration XML file.] ----------------------------------------------------------
Viewing All Prior Jobs
You can simply list the status of all jobs run also.
idracadm -r 10.1.8.59 -u root -p PASSWORD jobqueue view
-------------------------JOB QUEUE------------------------ [Job ID=JID_365611035667] Job Name=Configure: Import system configuration XML file Status=Completed Start Time=[Not Applicable] Expiration Time=[Not Applicable] Message=[SYS053: Successfully imported and applied system configuration XML file.] ---------------------------------------------------------- [Job ID=JID_254163911329] Job Name=Export: System configuration XML file Status=Completed Start Time=[Not Applicable] Expiration Time=[Not Applicable] Message=[SYS043: Successfully exported system configuration XML file.] ----------------------------------------------------------
Other Useful iDRAC Commands
You can reboot a bare-metal server remotely (useful for automating large-scale systems provisioning) via the following command:
idracadm -r 10.1.8.59 -u root -p PASSWORD serveraction powercycle
Rebooting a remote iDRAC.
idracadm -r 10.1.8.59 -u root -p PASSWORD racreset soft
You can also reboot a local iDRAC directly from the CLI of the server after querying it for information. This is useful in case of OOB failure/troubleshooting:
/opt/dell/srvadmin/sbin/srvadmin-services.sh start /opt/dell/srvadmin/sbin/racadm getsysinfo 2>&1 | less /opt/dell/srvadmin/sbin/racadm racreset
Updating the lifecycle controller firmware via racadm.
You can use an anon FTP server with the extracted payload/firmimg file. This can be extracted from the .EXE as it’s really just a zip archive. The example below is an iDRAC update with the 18.104.22.168 version renamed from payload/firmimg.d7
ssh mgmt-c10-h07-r730xd.example.com racadm fwupdate -f 10.12.64.163 ftp user -d pub/22.214.171.124.d7
Caveat – iDRAC username Length
We found a bug where if you exceed the username character limit of 16 characters the iDRAC will hard lock, requiring a manual hands-on reset or worse. We’ve filed this issue with Dell but aren’t sure if it’s fixed or not. Avoid usernames greater than 16 characters.
Caveat – iDRAC IP vs. FQDN
In later versions of the iDRAC firmware there’s a bug in DNS lookups that fail when using a FQDN with racadm to push commands. Use the IP address to work around this. We’ve traced this down via GDB and strace it and seems to fail getaddr() (we’ve also notified Dell).
Fix Hung iDRAC Tasks
Occassionally the iDRAC may get hung up on job requests not completing and piling up, rebooting will not fix this. The following command may work to unwedge them.
racadm -r drac.ip.address -u root -p password jobqueue delete -i JID_CLEARALL_FORCE
Then wait about 120 seconds and reboot the iDRAC again
racadm –r drac.ip.address –u root –p password racreset
Usage and Extension
We utilize the racadm CLI tools with a simple shell for loop to quickly provision multiple racks with only a few commands. We tie this into Foreman for full lifecycle deployment and management and tools like Ansible and Puppet to provide configuration management.
Using Ansible to Manage Dell Servers via Racadm
You can do a lot with Ansible, here’s some Ansible playbooks we use to do things like toggle NIC interface boot order or also change RAID configuration remotely and other things. Feel free to use/borrow these as examples and build upon them further.
When deploying new servers you can tie all this together. Everyones infrastructure is different but this is how we do it:
- Obtain MAC address information beforehand of any new servers and iDRAC
- Prep racadm commands with proper templating to fire off when servers are racked
- Prep your DHCPD entries for the iDRAC interfaces so they come online when powered on
- Prep the Foreman Hammer CLI commands to create host entries, set to build
- Once servers are powered on, run your deployment script
- racadm commands to create accounts, settings, carve disks etc.
- Foreman Hammer CLI commands to provision bare-metal hosts
- Optionally, automatically populate a Wiki based on the new servers
- If you used TRAC wiki, this rack generation tool might be useful
This set of examples is by no means exhaustive, experiment and see what else you can do with it. We’ve found almost everything you normally had to do within a browser or Java OOB can be done by referencing parameters in the iDRAC XML template. I take no responsibility for any damage that may be done. Please comment if you’ve got new use cases we can add.