Data Talks is hosting an SFTP server into which customers have access and can upload data in files.
However, there is some cases that the customers cannot automate the process of generating and uploading files to our SFTP server and we have to fetch the data instead from the SFTP server that the customers own.
We are developing the integration in which we can connect to any SFTP server and fetch the data that needs to be fetched on a scheduled basis.
One of the most popular methods of getting data from customers is through files. Customers create files that contain full snapshot or incremental data. Most preferred format of those files is the CSV one. Today we have developed code that tries to guess the parameters of the CSV format of the file like encoding, delimiter, escaping characters etc.
Although this method served well so far, we found out that this process may not be enough to cover for diversity of the configuration settings of the file. Thus, we are currently developing a feature to drive the process of discovering the settings of the CSV file by using metadata.
Υοu will be able to define the name of the file that will send over to us, together with a set of parameters that will describe how the file is delimited, how special characters are escaped and what is the encoding of the files.
At Data Talks we believe in transparency and we are planning to develop a set of features to increase the transparency even further. One of those features we recently launched is about internal customer data processing alerts.
The purpose is to increase the transparency about any potential issue that may affect the smooth customer experience. You as the customer do not need to act on the alerts, since our support team is on top of it and takes care of the issue.
As part of our product offering we have the possibility to add a very important feature that will assist our customers' decision making process and answer some important business questions. We call this feature Ad-hoc Analysis and a component of that feature is Exploratory Data Analysis & Insights.
Exploratory Data Analysis (EDA) is the process of analyzing and visualizing data to extract insights from it. Important characteristics of the data are revealed and the customer gets a better understanding of their data.
Exploratory Data Analysis & Insights is about:
• Increasing insight into a dataset
• Uncovering the underlying data structure
• Extracting important variables
• Detecting anomalies in data
• Testing assumptions
This is the process we are following:
1. Exploration of data. Looking for relationships between the data points. Exploring anomalies like duplicates, incorrect data types, missing data (nulls or empty strings) and any kind of suspicious observations. The results are communicated back to our customer to make sure that the data is correct and that there are no bugs causing the outliers.
2. Analyzing using statistical models. Employing statistical models and algorithms to analyze your data and get full insights.
3. Data storytelling. Presenting the findings back to the customer in a way of data storytelling, as well as having discussions with customers to answer business questions or find new data points to look closer at.
Data Cataloging gives us deep knowledge about our customers' data, not only in the Data Warehouse but in the Data Lake as well.Cataloging allows us to govern our Data Lake in a thorough manner and be on top of the data that exists in our Data Lake and where exactly that data is. What does this mean for you?
A Data Catalog can inform, and help prevent misuse of personal sensitive information, as well as provide documentation in case of an audit. In short, a Data Catalog provides a method to search and discover data in a way that is consistent with relevant regulations such as GDPR.
Data Profiling is the process of examining the data available in our Data Warehouse and collecting statistics and informative summaries about the data.
The main benefits of Data Profiling are:
1. Better data quality and credibility. Quality problems and anomalies in data are identified and communicated back to the customer.
2. Proactive crisis management. Profiled information enhances pro-activeness. Small mistakes in data can be stopped before they get bigger.
Below is a set of methods we use for our Data Profiling process:
1. Completeness Analysis. What is the ratio of empty, null and populated data in a field?
2. Uniqueness Analysis. How many unique values can be found in a given attribute across all records?
3. Range Analysis. What are the minimum, maximum, average and median values found for a given attribute?
With this feature you can filter on specific Segments (tags) in Data Talks PRO and then target the audience you want to reach using your preferred marketing automation tool (Salesmanago, in most cases). Basically, you decide who you want to reach using our CDP and then apply tags on this audience, which automatically will be reflected in your marketing automation tool.
For example, an ecommerce business can segment their customers based on the category of products (cameras, mobile phones, laptops etc) that are most purchased and on the brands customers prefer to buy. Then they can go into the marketing automation tool, filter on specific tags and run highly targeted marketing campaigns.
To be fully GDPR compliant we started implementing a set of techniques to reach the goal of high Data Privacy in our Platform. The latest techniques implemented are: Natural Key (ID) masking and separating sensitive and non-sensitive data.
What is Natural Key Masking? Natural Key Masking is the process of assigning a Natural Key of a physical person (customer) to a totally random (surrogate key). We developed this technique to ensure that if we are asked to delete all sensitive data about a customer, we still have the customer in our data models with an artificial ID that cannot point us to the actual physical person.
Note: Natural ID:s can sometimes be email addresses and National ID numbers. In these cases, we need to also delete the ID, but we make sure to keep the non-sensitive data connected to this person such as purchases and website visits.
Another feature that we introduced together with API Blackboxing is API Data to Platform. This means that we’re able to take care of the data as soon as it arrives at our API gateway so we can add it to your data model. What does this mean for you? It means that we can process the data to fit the business and technical requirements you need.
Our API is now upgraded with a new cool feature. Blackboxing allows us to buffer all incoming data in case there is something not functioning on the receiving side. This means that data will never be lost on the way to the end destination and you will be able to track what was sent to our API and when we received it. What does this mean for you?
More reliability.
We have now created a Sports Data Model that caters to the needs of the sports industry. This gives you as a sports customer the ability to manage the entire customer journey and get a full 360° view of your fans in one place. What does this mean for you?
You can now elicit top-notch insights about fans engagement, loyalty to the club, ticket sales, revenue figures plus more – and act on them in time for the next game.
