Cisco’s APIC-EM, or Application Policy Infrastructure Controller Enterprise Module is an OpenDayLight based SDN (Software Defined Network) controller. You could also possibly call it Cisco’s attempt to Merakify the Enterprise. On the bright side, it’s a free virtual appliance and no license is required.

One of the biggest features of APIC-EM is called Network Plug and Play

At a high level, the Cisco switch or router talks to the APIC-EM to streamline workflows and automate deployments. Switches and routers, known as agents discover the controller using any of the following mechanisms in order:

dhcp option 43 or dns (below), usb key, cloud discovery (currently beta), or the smartphone app.


option 43 ascii "5A1N;B2;K4;I172.19.45.222;J80"

The option 43 string has the following components, delimited by semicolons:

  • 5A1N;—Specifies the DHCP suboption for Plug and Play, active operation, version 1, no debug information. It is not necessary to change this part of the string.
  • B2;—IP address type:
    • B1 = hostname
    • B2 = IPv4 (default)
  • ;—IP address or hostname of the APIC-EM controller (following a capital letter i). In this example, the IP address is
  • Jxxxx —Port number to use to connect to the APIC-EM controller. In this example, the port number is 80. The default is port 80 for HTTP and port 443 for HTTPS.
  • K4;—Transport protocol to be used between the Cisco Plug and Play IOS Agent and the server:
    • K4 = HTTP (default)
    • K5 = HTTPS


APIC-EM: pnpserver.<customerdomain>.com
NTP Server: pnpntpserver.<customerdomain>.com

The DHCP pool will need to either be on vlan 1, or you’ll need to specify a staging vlan on the upstream switch:

pnp startup-vlan 55

That brings me to another caveat of of Plug and Play is that the firmware needs to be supported, and may not match the shipping version of the hardware!

Another feature of APIC-EM is called Easy QoS

I actually really like this use-case for network programmability. It’s important for the policies to match end-to-end in QoS, so being able to roll out policies and get insights into your policy-maps holistically is kind of a big deal.

APIC-EM documentation gives the concept of Northbound, which is the REST API you can use for custom applications, and Southbound in which APIC-EM talks to hardware using SNMP and CLI. Cisco states “future APIC-EM releases will leverage other southbound technology such as NetConf as they become available”.

I found some Postman collections from CiscoDevNet’s Github page here. Postman collections are a great way to learn by doing.

APIC-EM Firmware Compatibility

Official Getting Started Guide

You’ve probably heard a lot of buzzwords lately: API, JSON, and REST among them, however truth be told, REST or Representational State Transfer has always been around as the “language of the internet” (a REST HTTP GET brought you this page!), so it goes without saying that in the age of Internet of Things, REST would become infinitely more important.

REST is made up of 4 things

Request Type: POST, GET, PUT, etc
Header: Authorization (API key), Content-Type (html, application/xml, application/json)
Body: { “function” : “sendMessage”, “message” : “this is a message”}

JSON is the preferred format over XML because its more efficient. With PHP methods json_encode() and json_decode() for instance, it’s very easy to parse arrays into JSON format and vis a visa.

One of the great things about REST is that when a call is made, it sends a response, which can then in turn be a variable in a second response and so on. It’s not client-server, it’s a conversation.


Check out my demo’s for Spark API and Tropo API for a full writeup of how to write some simple REST calls using an html form and some PHP.

Cisco has a good writeup on CUC SQL queries, which are very useful to determine which voicemail accounts are active and which are just taking up a license, and perhaps could be skipped over during migrations. To get everything you need, use the query below:

lists all message counts, durations, size, and dates per alias:

run cuc dbquery unitymbxdb1 select alias as UserID, count (*) as TotalMessages, sum(case when deleted=’0′ then 1 else 0 end) as Inbox, sum(case when deleted=’1′ then 1 else 0 end) as Deleted,  min (arrivaltime) as OldestMessageTime, vw_mailbox.bytesize, sum(duration/1000) as TotalDuration_In_sec from vw_message,vw_mailbox, unitydirdb:vw_mailbox, unitydirdb:vw_user where vw_message.mailboxobjectid=vw_mailbox.mailboxobjectid and vw_mailbox.mailboxobjectid in (select mailboxid from vw_mailbox where unitydirdb:vw_user.objectid = unitydirdb:vw_mailbox.userobjectid) group by alias, vw_mailbox.bytesize order by TotalMessages desc


There are a number of deployment options for Expressway depending on your customers environment, which could lead you to having to come up with creative solutions. This is called engineering.

We know external services need to be discovered using “outside” dns, however what happens when a customer uses a single dns server for outside and inside networks? Well, The traversal zone will need to be a FQDN for TLS validation, so it needs to use dns to resolve the inside addresses, but clients on the public internet need to use dns to resolve the outside address of the Expressway E.


Tandberg VCS aka Cisco Expressway runs on linux and uses a lightweight network services package called dnsmasq

Being a linux guy, the first thing I tried was editing the /etc/hosts file, but to prevent hacking, everything is on a read-only filesystem except for the /tandberg mount, which also had an etc but changes were not persistent. After poking around I found the dnsmasq.conf man pages here and an option to specify a “conf-dir” to include configuration files that are loaded on start.

For static A records, create a file called hosts.conf



For static SRV records, create a file called srv.conf


After adding the file, you’ll need to restart dnsmasq:

/etc/init.d/dnsmasq restart


Adding to my post from monday, I had another issue with SRM not changing ip addresses on an Asterisk (CentOS) box, so I had to engineer a workaround..

For PRIMARY = and DR =

1. On the linux box, we’ll create two bash scripts, chmod these 755 and stick these in your $PATH.
ifconfig eth0 netmask up
route add default gw eth0
ifconfig eth0 netmask up
route add default gw eth0

2. On the vCenter, install WinSCP and run. Create 2 new sessions with saved credentials and name primary and dr. Connect to each session and accept the certificates!
3. Ensure the following is in the windows PATH, if not, add this directory: C:Program Files (x86)WinSCP
4. Create the batch file below to call winscp, start the session and run the relevant script from step 1.

@echo off
echo “Logging into host and running script”
If EXIST c:primary.txt (winscp primary /command “call”) ELSE (winscp dr /command “call”)
echo Exit Value %ERRORLEVEL%
TIMEOUT /t 5 /nobreak

In SRM run the bat file, no flags needed.

VMware advertises the ability to “perform fully automated orchestration of site failover and fallback with a single click.” While this technology is extremely easy to setup and use, it’s oftentimes the small things that cause the biggest headache. SRM uses dns to propogate ip changes from the primary to secondary virtual machines. With windows this is automatic: the servers simply register their new ip addresses in dns. For linux servers, however this is not so easy, especially in my case (an avaya phone system) which provides no access to the CLI. The fix is to create a script.

DNS Server –
Primary Site Avaya –
Failover Site Avaya-

First you’ll want to make sure you are running the SRM service as an account with domain admin rights (assuming you setup the database with windows authentication during setup).

Next you’ll need DNSCMD from the RSAT (remote server administration tools), which you can find here

Below is an example script:

@echo OFF
echo “Removing Records from DNS…”
dnscmd /recorddelete dom.local AVAYA-SRV-01 A /f
echo Exit Value:  %ERRORLEVEL%
TIMEOUT /t 3 /nobreak
dnscmd /recorddelete dom.local AVAYA-SRV-01 A /f
echo Exit Value:  %ERRORLEVEL%
TIMEOUT /t 3 /nobreak

@echo off
echo “Adding Records to DNS ..”
If EXIST c:primary.txt (dnscmd /recordadd dom.local AVAYA-SRV-01 A ELSE (dnscmd /recordadd dom.local AVAYA-SRV-01 A
echo Exit Value:  %ERRORLEVEL%
TIMEOUT /t 5 /nobreak

In SRM, you’ll call the script as:
C:windowssystem32cmd.exe /c c:srmscriptsdnschange.bat

Also, you may notice the “If EXIST c:primary.txt”
It was the most elegant way I could think of the check if the system was in a failover state. This file will exist only on the primary side. Credit to Derek Bickel on this one!

kron occurrence BACKUP in 24:00 recurring
policy-list Daily
kron policy-list Daily
cli show run | redirect tftp://


kron occurrence wBackup in 7:0:0 recurring
policy-list Daily
kron policy-list Daily
cli copy running-config flash:backupconfig.cfg