Marketing Segmentation & Predictive Analytics

Enterprises at any level need to target their consumers, clients, users with campaigns , measure the result of these campaigns and hopefully improve sales/contacts after each iteration. Practically this job requires a very broad spectrum of experiences starting from web/art/video design (how we format our message), going to language specific skills (how we write our message to raise interest) and of course some black magic art called segmentation (write the right message to the right potential or actual costumer). How marketing people build segments? Well using the attributes of the customers (age, sex, zip code, children etc…) they can create segments (for example all young women living in NY without children) and create campaigns for them (discounted shirts? Why not! ) . Now what is the role of predictive analytics in this area? Well in theory it should help marketing people really a lot: 

  • Discovering clusters (segments?) into the customer base
  • Identifying the key features or influencers that lead to buy an item or do some action
  • Showing what are the items  bought in combo and propose them as new packages to offer
  • Using social data identify leaders and followers into the customer base
  • ….surely  another hundred of insights like the ones mentioned.

Normally a marketing targeting tool leverages a classical database with tables, fields , records and actually the segmentation result it’s “just” a sql query that with some “where” conditions identifies the impacted customers. This tool it’s usually part of a suite, in other words from a vendor you buy a package with it and the package is made usually also of a CMS to build your web site pages, a eCommerce platform to sell your stuff on the net, an analytics/predictive analytics package to do the tasks described before. 

Looking at this , from a pure marketing department perspective, to maximize the value of your investments you should buy the package and enjoy the entire integrated platform without worries. It’s a bit like when you buy an iPhone and you start to enjoy it really when you couple it with an Apple TV, a MacBook and lately an apple watch. 

But while you can be the only one at home that takes this decision in medium/large enterprises you have several good reasons to do exactly the opposite:

1) predictive analytics runs on the largest possible combinations of data not only on the pure marketing/clicks/orders world.

2) it requires tools (big data clusters, in memory engines, complex algorithms, etc..) that are much more sophisticated  than the “normal” analytics capabilities provided by mkt packages . Usually it leverages best in class open source (R,Python,Scala libraries etc..) or specific vendor software (SAS, IBM,etc.. and lately several startups like Ayasdi etc..).

3) people working on it are miles away from marketing people from cab abilities perspective and from a target perspective (mkt usually want analytics to prove its guts feeling, analyst looks at data with curiosity trying to figure out patterns) 

4) From an analytics stand point  usually we want to buy software that we can reuse for all the business cases (supply chain, logistics, operations, etc…) and not only for the marketing business cases.

The usual “attempt to solve” this conflict it’s separation of diuties : analysts discover insights, they translate these insights into features of the customers and this info “should” fly to the segmentation tool and the good mkt guys should leverage these to do super effective campaigns. The result ? A huge amount of effort wasted without any ROI: 

1) mkt people do not own these features so they do not trust them or they simply ignore them

2) when you start the customization journey ( new interfaces that trasport data back & forth from analytics world to mkt planet) you will face bugs, data quality issues, data synchronization issues etc…

3) analyst try to cover multiple business cases as said, so only a fraction of his job it’s actually targeting marketing needs

When you are a startup however all the complications of this approach are mitigated by the fact that your team is small and mkt/analytics teams are in practice 2-3 people in the same room.

So we can apply the startup model everywhere?

The key factor here it’s people and objectives. We can go with a complete E2E marketing & predictive analytics integrated and customized solution but we have to add to it a dedicated team that works on it : creative marketing and data driven decisions can coexist and actually help each other if they are the result of a team of people laser focused on the same business objectives.

For large enterprises building this team is the key to provide  these capabilities as service to the various internal clients. Clients can be for example other marketing departments spread around the world in different countries working on different labels. These clients will benefit of this because they can focus really on their business at regional level and obtain not only an “agency like” service but they will benefit from the entire chain of experiences and results that data flowing at global level can bring.

    Integration with Adobe Marketing Cloud ( aka Neolane )

    Crm landscapes are often made of disparate systems interacting , for example you can have ,apart from the crm itself, marketing segmentation tools, data warehouses, e-Commerce sites, online presence sites,etc.. In this scenario one of the leading marketing cloud solutions is Adobe with the recent acquisition of Neolane. At very basics (Neolane experts please forgive me!) Neolane it’s a tool to segment your customers (young & low salary, old & high spending, single women with kids,etc….), create personalized campaigns for them and delivering to them commercial offers or discounts etc… With Neolane you can track the results of these campaigns (for example the clicked links on the campaign email) and optimize your next campaigns.

    From an integration perspective the classic way of import/export data with Neolane it’s the usage of files and workflows, but Neolane offers also other means to extract or update information on the system via API .

    The Neolane API consists of a single endpoint usually listening here :

    https://neolaneserver/nl/jsp/soaprouter.jsp

    This service accepts only POST requests and using standard SOAP message exchanges can help you to extract or change the data that you want on Neolane.

    However in order to access this resource you have to authenticate against Neolane. Once authenticated Neolane will give us a session token and a security token and using both we can finally query Neolane.

    Theoretically you can also query the soap service passing user/password combination but this security option is by default disabled at configuration level.

    For our purposes we will limit our analysis to these two soap services :

    1) session (Logon method)

    2) query (execute query method)

    Now in order to proceed to the API calls we can proceed in two ways:

    A) call this page https://neolaneserver/nl/jsp/schemawsdl.jsp?schema= for each service and passing the service name as parameter obtain the wsdl (you have to authenticate so go with your browser and save the wsdl to disk). For session you will make the call with this parameter xtk:session  , for execute query with this parameter xtk:queryDef. Once you have the wsdl you can use the tool of your choice to generate the classes for the integration (for .net wsdl.exe).

    B) instead of having proxy classes that handle for you the burden of soap communication you can call directly https://neolaneserver/nl/jsp/soaprouter.jsp and modify the request headers according to the Soap Action (service) that you want to call.

    We will go with option B for two reasons:

    1) the authentication service (session) requires credentials passed inside the http header of the request and do this with .net it’s pretty cumbersome and involves the modification of the proxy classes (you will lose this if you generate again them against the wsdl)

    2) the proxy classes generated , in my case at least, are not really that good … just to give you an example :
    This the wsdl section for the Logon method of Session Service

     <s:element name="Logon">
      <s:complexType>
        <s:sequence>
           <s:element maxOccurs="1" minOccurs="1" name="sessiontoken" type="s:string"/>
           <s:element maxOccurs="1" minOccurs="1" name="strLogin" type="s:string"/>
           <s:element maxOccurs="1" minOccurs="1" name="strPassword" type="s:string"/>
           <s:element maxOccurs="1" minOccurs="1" name="elemParameters" type="tns:Element"/>
        </s:sequence>
      </s:complexType>
     </s:element>
      <s:element name="LogonResponse">
       <s:complexType>
         <s:sequence>
          <s:element maxOccurs="1" minOccurs="1" name="pstrSessionToken" type="s:string"/>
          <s:element maxOccurs="1" minOccurs="1" name="pSessionInfo" type="tns:Element"/>
          <s:element maxOccurs="1" minOccurs="1" name="pstrSecurityToken" type="s:string"/>
         </s:sequence>
       </s:complexType>
    </s:element>
    
    

    and this is the generated method

    [System.Web.Services.Protocols.SoapDocumentMethodAttribute("xtk:session#Logon", RequestNamespace="urn:xtk:session", ResponseNamespace="urn:xtk:session", Use=System.Web.Services.Description.SoapBindingUse.Literal, ParameterStyle=System.Web.Services.Protocols.SoapParameterStyle.Wrapped)]
        [return: System.Xml.Serialization.XmlElementAttribute("pstrSessionToken")]
    public string Logon(string sessiontoken, string strLogin, string strPassword, System.Xml.XmlElement elemParameters, out System.Xml.XmlElement pSessionInfo, out string pstrSecurityToken) {
            object[] results = this.Invoke("Logon", new object[] {
                        sessiontoken,
                        strLogin,
                        strPassword,
                        elemParameters});
            pSessionInfo = ((System.Xml.XmlElement)(results[1]));
            pstrSecurityToken = ((string)(results[2]));
            return ((string)(results[0]));
        }
    

    so basically it requires you to pass parameters by reference (out keyword) while in reality there is no need for it.

    So let’s go with classic HttpWebRequest:

     //Create the web request to the soaprouter page
    HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://neolaneserver/nl/jsp/soaprouter.jsp");
    
    req.Method = "POST";
    req.ContentType = "text/xml; charset=utf-8";
    //Add to the headers the requested Service (session) that we want to call
    req.Headers.Add("SOAPAction", "xtk:session#Logon");
    //Here for testing purpouses username and password are here but you should acquire it in a secure way!
    string userName = "username";
    string pass = "password";
    //We craft the soap envelope creating a session Logon reequest
    string body = "<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:urn=\"urn:xtk:session\">" +
                    "<soapenv:Header/><soapenv:Body><urn:Logon>" +
                    "<urn:sessiontoken/>" +
                    "<urn:strLogin>" + userName + "</urn:strLogin>" +
                    "<urn:strPassword>" + pass + "</urn:strPassword>" +
                    "<urn:elemParameters/>" +
                "</urn:Logon></soapenv:Body></soapenv:Envelope>";
    
    //We write the body to a byteArray to be passed with the Request Stream
    byte[] byteArray = Encoding.UTF8.GetBytes(body);
    
    // Set the ContentLength property of the WebRequest.
    req.ContentLength = byteArray.Length;
    // Get the request stream.
    Stream dataStreamInput = req.GetRequestStream();
    // Write the data to the request stream.
    dataStreamInput.Write(byteArray, 0, byteArray.Length);
    // Close the Stream object.
    dataStreamInput.Close();
    
    var response = req.GetResponse();
    
    Stream dataStream = response.GetResponseStream();
    // Open the stream using a StreamReader for easy access.
    StreamReader reader = new StreamReader(dataStream);
    // Read the content.
    string responseFromServer = reader.ReadToEnd();
    // Display the content on the console.
    Console.Write(responseFromServer);
    // Clean up the streams and the response.
    reader.Close();
    response.Close();
    
    //Manually parsing the response with an XMLDoc
    System.Xml.XmlDocument xResponse = new XmlDocument();
    xResponse.LoadXml(responseFromServer);
    // We parse manually the response. This is again for testing purpouses
    XmlNode respx = xResponse.DocumentElement.FirstChild.FirstChild;
    
    string sessionToken = respx.FirstChild.InnerText;
    string securityToken = respx.LastChild.InnerText;
    // We have done the login now we can actually do a query on Neolane
    HttpWebRequest reqData = (HttpWebRequest)WebRequest.Create("https://neolaneserver/nl/jsp/soaprouter.jsp");
    reqData.ContentType = "text/xml; charset=utf-8";
    //Add to the headers the requested Service (ExecuteQuery) that we want to call
    reqData.Headers.Add("SOAPAction", "xtk:queryDef#ExecuteQuery");
    //Add to the headers the security and session token
    reqData.Headers.Add("X-Security-Token", securityToken);
    reqData.Headers.Add("cookie", "__sessiontoken=" + sessionToken);
    reqData.Method = "POST";
    // We write a SQL Like query to Neolane using XML syntax. Basically we are asking: SELECT email,firstname,lastname from recipient where email='test@test.com'
    string bodyData = "<soapenv:Envelope xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:urn=\"urn:xtk:queryDef\">" +
                    "<soapenv:Header/><soapenv:Body><urn:ExecuteQuery><urn:sessiontoken/><urn:entity>" +
                    "<queryDef operation=\"select\" schema=\"nms:recipient\">" +
                        "<select><node expr=\"@email\"/><node expr=\"@lastName\"/><node expr=\"@firstName\"/></select>" +
                        "<where><condition expr=\"@email = 'test@test.com'\"/></where>" +
                    "</queryDef>" +
                "</urn:entity></urn:ExecuteQuery></soapenv:Body></soapenv:Envelope>";
    
    byte[] byteArrayData = Encoding.UTF8.GetBytes(bodyData);
    
    // Set the ContentLength property of the WebRequest.
    reqData.ContentLength = byteArrayData.Length;
    // Get the request stream.
    Stream dataStreamInputData = reqData.GetRequestStream();
    // Write the data to the request stream.
    dataStreamInputData.Write(byteArrayData, 0, byteArrayData.Length);
    // Close the Stream object.
    dataStreamInputData.Close();
    
    var responseData = reqData.GetResponse();
    
    Stream dataStreamData = responseData.GetResponseStream();
    // Open the stream using a StreamReader for easy access.
    StreamReader readerData = new StreamReader(dataStreamData);
    // Read the content.
    string responseFromServerData = readerData.ReadToEnd();
    // Display the content. Here we will see the query results in form of recipient collection
    Console.Write(responseFromServerData);
    // Clean up the streams and the response.
    readerData.Close();
    responseData.Close();
    

    Using this technique you can simply pull or push data to Neolane without having to touch the Neolane system itself.

    UPDATE : here you can see how to write to Neolane!

    DIY port scanner for windows sysadmins 

    I have recently been involved in a work of security assessment on multiple servers hosted in the same data center but in different vlans. One requirement that I had to respect it is that the port scan procedure on all these servers (all windows 2008 servers) should be fast and not require additional software to buy . Unluckily the tool that was used was a simple netstat command to be issued manually on each server.

    Now this has the advantage of being fast (netstat gives you the overall situation in a fraction of second) but has at least two major limitations : no indication of the process running on the port (only the PID) and most of all is completely manual.

    First thing I tried was to leverage powershell commands , specifically this great resource that basically gives already the right process name and other useful info on the processes running on each port with all the required network statistics. Theoretically this should work also remotely but I never managed to make it work successfully (several times crashing because of the need of writing on the remote  system a file). So I thought that netstat and a good combination of great Mark Russinovich PSTools (psexec to execute netstat remotely and pslist to retrive the processes remotely) could solve the problem. In theory yes, in practice I had to fight a bit with c# to obtain the desired result.

    Here the recipe:

    First go here to see how to launch PSTools from c# and parse the output coming from netstat. Be aware that this does not work on the local computer but you have to issue the netstat without psexec.

    Second apply the same trick to pslist and parse the results (this works also on the local computer).

    Third now that you have this just put the data inside some structures (I used DataTables) and with a simple Linq query you will join network statistics and processes running.

    The only thing to check is that PsTools works requiring that Ports 135 and 445 (TCP) need to be open and Admin$ and IPC$ shares enabled, so in my case I had to run my console program on each segregated VLAN and in some cases on some servers alone, but finally I managed to cut the manual operations from 75 manual runs to 8 and it’s already a good result.

    Crack .NET assemblies on the fly

    Hi, I recently have been into a DEFCON 1 situation and I want to share with you my experience.

    Basically imagine a production .net assembly that does the job quietly for 5-6 years and one day it decides to stop working ,this means that an entire segment of the business of a large enterprise is completely blocked…

    To this add that developer of that assembly no longer works in the company, nobody knows where the source code can be and actually there is no  documentation.

    So they call me and I felt like this:

    So downloaded ILSpy and I started looking into the code while gathering the logs and checking the exceptions.

    Thanks to some very lucky combo of checks I was able to figure out where the problem was in a couple of hours (windows update I hate you), but in the mean time the pressure bar was rising and the client was just going crazy. So I remembered that one time I used a tool called reflexil to fix on the fly an assembly, but later I figured out that it works nice with reflector but not with IlSpy…

    However thanks to this post I was able to download a version of reflexil that was compatible with IlSpy and problem solved!

    Here is a screenshot of the two working together :

    ILSpyReflexil

    It’s actually a great combo and I can’t wait to have the final bits.

    PS : The code that you see it’s not the client’s code of course 🙂

    Here I am…

    start

    Hi to everyone, according to blog guides in this first post I should describe my self and why users should read my blog. In reality I just needed a place where to write freely about my experiences with different products/technologies.

    You can find here security related articles, specific Microsoft technology posts , some experiences with some Big Data products/platforms and also some other collateral stuff.

    Sometimes I loved that stuff , some other I hated with all my self, but in the end in both cases the important thing it’s the experience coming from it.

    Several times even if I was working on something that was non particularly interesting I discovered precious info that it became really useful later on other projects.

    Of course it also happens the opposite , working on something nice and you discover things that will became later very useful .

    Just an example that comes to my mind :

    I was reading for personal interest Stefan Esser findings on IOS jailbreak and from that I figured out a bit some strategies that Apple uses to handle kernel memory and this became fundamental later when I started to help a team that was struggling to design a decent protection on their app from memory dumps.

    My primary objective with this it’s not to showcase something but receive primarily a feedback on some activities that I’m doing .

    Good or bad feedback does not matter, it’s just interesting to hear multiple voices and ideas.