Organisations are beginning to ‘listen’ to unstructured data found in texts which was previously deemed boring or irrelevant to provide insights, says Evan Harridge, founder of Immersive.

Immersive uses text analytics via machine learning to uncover new insights from unstructured data and has developed a ‘sentiment index’ which allows companies to discern how happy or unhappy customers are based on the content of their emails.

“In a lot of cases we are looking at opportunities that don’t exist, because we can now store and analyse all of this information which was just seen as useless or not valuable,” Harridge told Which-50 during The Hadoop Summit in Melbourne last week.

“Customers are starting to realise all the conversations and all the email communications we have potentially generate value. Rather than just the summarised information or the headings or the information inside the cost table, we take everything and use it as context.”

Immersive worked with the Victorian Department of Justice to secure text information and allow 10,000 workers in Victoria to find information quickly across multiple systems using text search and indexing technologies. The company has also examined text reports from the Fire Services Commissioner to identify any trends behind the cause of fires.

Immersive worked with real estate giant JLL to build technology that uses machine learning to read leases and extract potential risks or potential break clauses — all of the elements a legal clerk or lawyer would look for — and generate alerts.

Harridge said Immersive also looks at vast swathes of email information to generate value.

“We look at the tone and timbre of email interactions staff have with their customers and we look for any opportunities for recommendations or alerts if there is any discussion around a piece of business being lost or maybe around an opportunity for a new product or process to be sold,” Harridge said.

“We like to build a tool so you can watch all of these interactions in real time.”

Harridge sees the future being increasingly impacted by insights driven by machine learning which can inform real time decisions, predictions and recommendations.

“We are already doing some experiments with chatbots generating in-context recommendations and allowing someone to use a message to solve a problem rather than having to go to an interface and use a navigation method to find a function. We see that as the future, we are spending a lot of time and energy researching that.”

Immersive uses open source software framework Hadoop to store the information it collects.

“Often you don’t know what you are looking for. You just need to store it first and then the insight you are looking for reveals itself. It’s very much the toolkit that allows us to store and build a lot of these interesting applications,” Harridge said.

IT In Reverse
Scott Gnau, CTO Hortonworks, describes this process as IT in reverse. Traditionally an IT department would start with the business requirements, find the data and build an application. That process has now changed direction.

“In the old world you started with the requirements and go find the data,” Gnau said. “In the new world you start with the data and you go find the requirements. It’s IT in reverse. It’s letting the data tell you things about your business and not your business tell you things about your data.”

This change in process has implications for technology and thinking for IT departments, Gnau said.

“For the last 30 years it was all about converged systems, centralised systems, bringing data in and normalising data… That centralisation or converging of systems was enabling some cost savings and some good analytics,” Gnau said.

“Data is now going to be decentralised for a very long time. That has implications on how technology needs to behave.”

Gnau argued data platforms should now be connected, rather than converged, which means moving away from a single integrated stack provider to an ecosystem and integration of multiple technologies.