Insight for Non-Programmers

In order to gain insight from data, there are several possible ways to take. 4 main methods are explained here, which range from starting with few resources to month long projects. The information is meant to give you some background on what you can do as a non-programmer to ensure you get the insight you want.

4 Methods of Exploration

Spreadsheet

Advanced

BI-Solution

AI or ML

This is the easiest and most inexpensive solution. Get data from tech and work on it with your classic spreadsheet application.

This more advanced way of analyzing data uses specialized software like Tableau. While it presents results better, it will take weeks to set up.

The large commercial BI solutions from SAP or Microsoft offer great benefits, but come with longer implemention times and costs.

Instead of looking yourself, why not let artificial intelligence or simpler machine learning algorithms look for new insight. Offers like Salesforce Einstein promise to do just that.

Spreadsheet Exploration

Pure data records are good for computers to handle, but make it almost impossible for humans to comprehend. It is therefore necessary to filter, bundle or transform the data in some way to make it understandable.

In order to do that, the technical side will need some sort of information on what to deliver from the business side. If you work on the business side of things, it is useful to "translate" your request into a more technical language. This makes it easier for programmers or data scientists to understand it. Here are some ideas on how to do that:

 In most cases you want to do your own data exploration of given data. For example, take the times and amounts of sales in web shop. You can yourself find mean values or medians for your data or filter it so you have just evening sales. You should therefore have some experience using spreadsheets or ask your technician to implement some basic functions in them. Furthermore, you should ask for a "good" amount of data for you to explore. Just a hundred data points might be too few and a million to big for you to handle. It sounds simple, but ask for a sound description of the data you receive. Technically, this is usually called metadata. If you know where a column of data come from, it is a lot easier to extend your search into that area later. Also talk about timeliness of the data you receive. Sometimes your are happy with data that is a month old, but usually you want current data. Maybe you can receive updated data in certain intervals. Finally, talk about how empty or erroneous data is treated. If half of the data is left out of the subset you get, you might have a wrong perception of the underlying data. Some queries leave out the whole data point, where there is just one data cell missing. But in many cases this is optional data, like a second phone number. If it is hard to get all the data, at least get numbers on how much data was left out of what you got.

Get more information on this informative page from Microsoft.

Checklist Spreadsheet Exploration

  • Ask for the data set in spreadsheet format
  • Get good amount of data points (couple thousand)
  • Ask for description of data (Metadata)
  • Get current data or regular updates
  • Ask about empty or erroneous data points in your set

Advanced Exploration

Once you have analyzed data with standard spreadsheet capabilities a couple of times, you usually find patterns that you would like to review regularly. Both Tableau and MS Power BI offer you to visualize data repeatedly in different ways. This allows you build great looking visualizations to base your discussions on.

The two tools mentioned above have their main focus on making it easy to present data visually and not so much on how to get the data in the right format beforehand. Tools like pentaho are better suited for this, but they require many assumptions in order to process the data. Sales data could, for example, include empty values for the price as they were paid in a foreign currency, which is stored in a different table or data set. This could lead to drawing wrong conclusions when using an automated data extraction. Whereas you get the data already in a usable form from a technician, they will usually have the knowledge that this sales data is stored in different places.

Checklist Advanced Exploration

  • Ask for automated updates of the data (e.g. every hour)
  • Think of the best way represent differences as you will get data regularly
  • Ask for the data to be transformed in the same way every time
  • Think of the right moment to upgrade to a BI solution

Business Intelligence (BI) solution

A full business intelligence solution will cost substantial resources to introduce and should therefore be implemented in a bigger project.
In my experience, it is sufficient to us whatever possibilities your current system is offering to get data out and then visualize it with either one of the methods presented above. If you introduce a BI solution, oftentimes the carefully introduced outputs are already outdated once they are introduced. This doesn't hold true for all situations, but if you know exactly what to look for, you can usually find that with the tools you already have. If you want fresh insight, you will need to try new ideas fast and easily and this is usually not the stronghold of bigger BI solutions.

Artificial Intelligence (AI) or Machine Learning (ML)

The terms artificial intelligence and machine learning are very commonplace today and are also used in the space of deriving insight from data. The idea is, that insight will come automatically from the software. Currently, there is no such software on the market and it seems unlikely to be available soon. The current hype surrounding the subject is a much improved ecosystem of tools and combination of techniques, that make classic methods of analysis quicker and cheaper to implement. Read more about that in this post.