Bot hand off to agent with Salesforce Live Chat Part 2

Hi our previous article we introduced the api calls to send and receive messages to a live agent on Salesforce. Now it’s time to add the bot component and combine bot and live agent to implement the hand off .

For the bot I used one the of frameworks I know better the Microsoft Bot Framework , but some of the concepts can be applied also to other bot solutions.

We start using the Bot Intermediator Sample provided here  , that has already some functionality built in. In particular it uses the bot routing engine that can has been built with the idea of routing conversations between user, bot and agent , creating when needed direct conversations between the user and agent that is actually routed by the bot using this engine.

Let’s see a way that we can use to combine this with salesforce live agent api , we will take some shortcuts and this solution it is not meant to be used in production environment, but hopefully can give you an idea of how you can design a fully fledged solution .

  1. When in the conversation is mentioned the word “human” the intermediator sample triggers the request of intervention of an agent and parks the request inside the database of pending requests of the routing engine . Our addition it has been to define an additional ConcurrentDictionary as in memory storage to store the request and its conversation and add later other properties interesting for us.
  2. Using quartz scheduling engine we can monitor with a recurring job the pending requests of the routing engine , dequeue them starting (always using quartz) an on demand job that opens a connection with live chat , waits that the agent takes the call and binds into to the request the sessionId and the other properties of the LiveChat session opened. This thread can finish here but before we start another on demand thread that is watching any incoming message coming for this request from LiveChat session and routes them to the conversation opened at step 1
  3. In the message controller of the bot, in addition to the default routing rules, we add another rule that checks if the current conversation is “attached” to a live chat session and if yes sends all the chat messages written by the user to the related live chat session.
  4. When the watch live chat session thread does not receive more messages goes in timeout or receives a disconnect/end chat event , it removes the conversation with live chat session from the dictionary and from this moment if the user writes again , he will write to the bot and he wants again to speak with an agent he has to trigger the human “keyword” again.

Here some screenshots:

Chat begins with bot that simply repeats the sentences we write

Screen Shot 2018-03-20 at 9.32.05 PM

Live Agent is ready to handle new calls

Screen Shot 2018-03-20 at 9.32.27 PM


Let’s ask for help

Screen Shot 2018-03-20 at 9.35.01 PM

And here the request arrives on live chat

Screen Shot 2018-03-20 at 9.35.15 PM

Once accepted we can start the hand off starting a case in salesforce

Screen Shot 2018-03-20 at 9.35.28 PM

And here we can check if we are taking to a human 🙂

Screen Shot 2018-03-20 at 9.38.56 PM

Screen Shot 2018-03-20 at 9.38.40 PM

In the third and final part we will look inside some code snipplets that show case this functionality and we will describe what can be a good design of the solution if we want to industrialize it.



Bot hand off to agent with Salesforce Live Chat Part 1

Hi everyone, one of the most requested features into modern implementations is a smooth transition from the automated response system (our lovely bot) to a human.

Our objective in fact is usually the following:

  1. Handle the customer request  first doing a qualification of the request (collect data, ask additional information)
  2. Now it can happen that the request can be handled with simple and repetitive solution and bot should exactly cover this scenario
  3. It can also happen that the request is so complex that can be handled only by a call center operator but we will make good usage of the operator’s time because he will be involved in an activity where he can bring a distinctive value

One the most used Call Center modules for human assistance on a case is Salesforce Live Chat and it makes sense to understand how we can make a transition from any bot implementation to Live Chat without requesting the customer to change UI, transition to another web page and more importantly to re-type all the information he wrote at the qualification state (so assuming that the triage has been done in the bot application we want to bring the entire conversation state from the bot to the live agent attention).


Let’s start with the basics and see the “how to” from the beginning:

First you need a salesforce developer sandbox for your testing , you can request one for free here.

Once you have your sandbox you have to enable the live agent functionality, following the steps described here , please pay attention to each step and your last step should be this one .

You can try if everything works just creating a sample html page with javascript created by the buttons functionality and the deployment one (remember to put the deployment javascript at the end of the page before the closing body tag!).

If you want an unofficial guide to help you more check also this blog  or this other blog .

At this point you should have your live chat working nicely and we can now proceed to study the salesforce live agent rest api that allow us to us the live chat functionality programmatically.

If you look a bit to how the API works you will soon notice that this API has been design to be consumed mainly directly by final clients (web pages or mobile apps) while it lacks some Server to Server functionality like web-hooks , so in a nutshell it is very helpful if you want to build a branded web page or IOS/Android app for call center support but it a bit less helpful to use it for transitioning a conversation from a server application (our bot).

In order to use the api we need some info: your Salesforce Organization Id ,  your live agent deploymentId , live agent buttonId and finally the live agent api endpoint.

You can find this info here and in this guide.

Ok now can finally start with some coding 🙂 , I will use c# (running from a Mac) so I guess it can run on any platform .

First we need to do our first rest call to retrive the session ID for the new session, the session key for the new session, the affinity token for the session that’s passed in the header for all future requests and finally the clientPollTimeout that represents the number of seconds before you must make a Messages request before your Messages long polling loop times out and is terminated (we will understand this better later):

 private static async Task<ChatObj> createSession()
             string sessionEndpoint = liveAgentEndPoint + liveAgentSessionRelativePath;
             HttpClient client = new HttpClient();
         client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
             client.DefaultRequestHeaders.Add("X-LIVEAGENT-API-VERSION", liveAgentApiVersion);
             client.DefaultRequestHeaders.Add("X-LIVEAGENT-AFFINITY", "null");
             HttpResponseMessage response = await client.GetAsync(sessionEndpoint);
             JObject jObj = new JObject();
             if (response.IsSuccessStatusCode)
                 string resp = await response.Content.ReadAsStringAsync();
                 jObj = JObject.Parse(resp);

             ChatObj chatObj = new ChatObj();
             return chatObj;

Now that we have this information we can actually say to the live agent that we would like to start a chat session with him (!) and this requires another api call to request a chat visitor session and this session will be actually opened only when the live agent accepts the request into the salesforce console.

So first we do the request:

  private static async Task createChatRequest(ChatBag chatObj)
             HttpClient client = new HttpClient();
             client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
             client.DefaultRequestHeaders.Add("X-LIVEAGENT-API-VERSION", liveAgentApiVersion);
             client.DefaultRequestHeaders.Add("X-LIVEAGENT-AFFINITY", chatObj.getAffinityToken());
             client.DefaultRequestHeaders.Add("X-LIVEAGENT-SESSION-KEY", chatObj.getSessionKey());
             client.DefaultRequestHeaders.Add("X-LIVEAGENT-SEQUENCE", "1");
             JObject body = new JObject();
             body.Add(new JProperty("organizationId", liveAgentOrgId));
             body.Add(new JProperty("deploymentId", liveAgentDeploymentId));
             body.Add(new JProperty("buttonId", liveAgentButtonId));
             body.Add(new JProperty("sessionId", chatObj.getSessionId()));
             body.Add(new JProperty("trackingId", ""));
             body.Add(new JProperty("userAgent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36"));
             body.Add(new JProperty("language", "en-US"));
             body.Add(new JProperty("screenResolution", "1440x900"));
             body.Add(new JProperty("visitorName", "ConsoleTest"));
             body.Add(new JProperty("prechatDetails", new List<String>()));
             body.Add(new JProperty("receiveQueueUpdates", true));
             body.Add(new JProperty("prechatEntities", new List<String>()));
             body.Add(new JProperty("isPost", true));
             StringContent cnt = new StringContent(body.ToString(), Encoding.UTF8, "application/json");
             HttpResponseMessage response = await client.PostAsync(liveAgentEndPoint + liveAgentChasitorRelativePath, cnt);
             if (response.IsSuccessStatusCode)
                 string responseText = await response.Content.ReadAsStringAsync();


If everything went right we should receive an “OK” as response while we wait for the operator to actually accept the visitor session request.

An important thing to notice is that the API supports prechatDetails and prechatEntities objects that we can use to bring with us the conversation data that the customer had with the bot , so the live agent can look at this info and immediately help the customer with the right context without re-asking the same questions.

Since the process of approval to start the chat is not automatic but we have to wait for the live agent to accept, at this stage we have just to poll the Message api and wait for having the confirmation using a thread that calls the api in this way:

  private static async Task<ChatMessageResponse> receiveMessages(ChatBag chatObj)
             ChatMessageResponse jObj = new ChatMessageResponse();
             HttpClient client = new HttpClient();
             client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
             client.DefaultRequestHeaders.Add("X-LIVEAGENT-API-VERSION", liveAgentApiVersion);
             client.DefaultRequestHeaders.Add("X-LIVEAGENT-AFFINITY", chatObj.getAffinityToken());
             client.DefaultRequestHeaders.Add("X-LIVEAGENT-SESSION-KEY", chatObj.getSessionKey());

            HttpResponseMessage response = await client.GetAsync(liveAgentEndPoint + liveAgentMessagesRelativePath);
             if (response.IsSuccessStatusCode)
                 string respText = await responseContent.ReadAsStringAsync();
                 jObj = JsonConvert.DeserializeObject<ChatMessageResponse>(respText);

                 if (jObj!=null)
                     var msgs = from x in jObj.messages
                                                             where x.type == "ChatRequestSuccess"
                                    select x;
                     foreach (Messages activity in msgs)
                         Console.WriteLine("VisitorId: " +activity.message.visitorId);
             return jObj;

Ok so when we receive the ChatRequestSuccess Type message, this means that chat request was successful and routed to available agents .

To be completely sure that an agent really accepted our conversation we have to wait for the ChatEnstablished Type message where we can also read the name and the id of the agent answering us.

Ok now we can finally send an “Hello Mr Agent!” text to our Live Agent with this api:

  private static async Task sendTxtMessage(ChatBag chatObj,string textToSend)
             HttpClient client = new HttpClient();
             client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
             client.DefaultRequestHeaders.Add("X-LIVEAGENT-API-VERSION", liveAgentApiVersion);
             client.DefaultRequestHeaders.Add("X-LIVEAGENT-AFFINITY", chatObj.getAffinityToken());
             client.DefaultRequestHeaders.Add("X-LIVEAGENT-SESSION-KEY", chatObj.getSessionKey());
             JObject bodyT = new JObject();
             bodyT.Add(new JProperty("text", textToSend));
             StringContent cnt = new StringContent(bodyT.ToString(), Encoding.UTF8, "application/json");
             HttpResponseMessage response = await client.PostAsync(liveAgentEndPoint + liveAgentChasitorChatRelativePath, cnt);
             if (response.IsSuccessStatusCode)
                 string respText = await responseStep2.Content.ReadAsStringAsync();

And we can receive the replies of the agent always using the same receive message polling technique but this time searching for  ChatMessage  type kind of messages.

In the next part of the article I will go through the integration with a bot and attempt to see how we can implement the hand off !

Jazoon 2017 AI meet Developers Conference Review

Hi I had the opportunity to participate to this conference in Zurich on the 27 October 2017 and attend to the following sessions:

  • Build Your Intelligent Enterprise with SAP Machine Learning
  • Applied AI: Real-World Use Cases for Microsoft’s Azure Cognitive Services
  • Run Deep Learning models in the browser with JavaScript and ConvNetJS
  • Using messaging and AI to build novel user interfaces for work
  • JVM based DeepLearning on IoT data with Apache Spark
  • Apache Spark for Machine Learning on Large Data Sets
  • Anatomy of an open source voice assistant
  • Building products with TensorFlow

Most of the sessions have been recorded and they are available here:

The first session has been a more a sales/pre-recorded demos presentation of SAP capabilities in terms of AI mainly in their cloud:


But with some interesting ideas like the Brand Impact Video analyzer that computes how much airtime is filled by specific brands inside a video:


And another good use case representation is the defective product automatic recognition using image similarity distance API:


The second session has been around the new AI capabilities offered by Microsoft and divided into two parts:

Capabilities for data scientists that want to build their python models

  • Azure Machine Learning Workbench that is an electron based desktop app that mainly accelerates the data preparation tasks using “a learn by example” engine that creates on the fly data preparation code.


  • Azure Notebooks a free but limited Cloud Based Jupyter Notebook environment to share and re-use models/notebooks


  • Azure Data Science Virtual Machine a pre-built VM with all the most common DS packages (TensorFlow, Caffe, R, Python, etc..)


Capabilities (i.e. Face/Age/Sentiment/OCR/Hand written detection) for developers that want to consume Microsoft pre-trained models calling directly Microsoft Cognitive API



The third session has been more an “educational presentation” around deep learning, and how at high level a deep learning system work, however we have seen in this talk some interesting topics:

  • The existence of several pre-trained models that can be used as is especially for featurization purposes and/or for transfer learning


  • How to visualize neural networks with web sites like
  • A significant amount of demos that can show case DNN applications that can run directly in the browser

The fourth session has been one also an interesting session, because the speaker clearly explained the current possibilities and limits of the current application development landscape and in particular of the enterprise bots.


Key take away: Bots are far from being smart and people don’t want to type text.

Suggested approach bots are new apps that are reaching their “customers” in the channels that they already use (slack for example) and those new apps using the context and channel functionalities have to extend and at the same time simplify the IT landscape.


Example: bot in a slack channel that notifies manager of an approval request and the manager can approve/deny directly in slack without leaving the app.

The fourth and the fifth talk have been rather technical/educational on specific frameworks (IBM System ML for Spark) and on models portability (PMML) with some good points around hyper parameter tuning using a spark cluster in iterative mode and DNN auto encoders.



The sixth talk has been about the open source voice assistant MyCroft and the related open source device schemas.

The session has been principally made on live demos showcasing several open source libraries that can be used to create a device with Alexa like capabilities:

  • Pocketsphinx for speechrecognition
  • Padatious for NLP intent detection
  • Mimic for text to speech
  • Adapt Intent parser


The last session was on tensor flow but also in general experiences around AI coming from Google, like how ML is used today:


And how Machine Learning is fundamental today with quotes like this:

  • Remember in 2010, when the hype was mobile-first? Hype was right. Machine Learning is similarly hyped now. Don’t get left behind
  • You must consider the user journey, the entire system. If users touch multiple components to solve a problem, transition must be seamless

Other pieces of advice where around talent research and maintain/grow/spread ML inside your organization :

How to hire ML experts:

  1. don’t ask a Quant to figure out your business model
  2. design autonomy
  3. $$$ for compute & data acquisition
  4. Never done!

How to Grow ML practice:

  1. Find ML Ninja (SWE + PM)
  2. Do Project incubation
  3. Do ML office hours / consulting

How to spread the knowledge:

  1. Build ML guidelines
  2. Perform internal training
  3. Do open sourcing

And on ML algorithms project prioritization and execution:

  1. Pick algorithms based on the success metrics & data you can get
  2. Pick a simple one and invest 50% of time into building quality evaluation of the model
  3. Build an experiment framework for eval & release process
  4. Feedback loop

Overall the quality has been good even if I was really disappointed to discover in the morning that one the most interesting session (with the legendary George Hotz!) has been cancelled.