Analytics are for everyone !

Analytics are for everyone! Well, not building analytics, no. That needs a high level of expertise in statistics, machine-learning, optimization, programming, database skills, a healthy does of domain knowledge for the problem being addressed and a pretty wide masochistic streak too.

Using analytics, now that is for everyone, or at least it should be. We all use analytics, and, I think, the best examples, we use without thinking about just how complex it is.

Is there anyone out there that hasn't used an electronic mapping service (GPS) for directions? Even ignoring the electronics, these are remarkable pieces of engineering! An extensive, detailed database of road systems and advanced routing analytics to help you find the best route from A to B without sending you backwards down one-way roads or across half-finished bridges.

Perhaps you're thinking it's not that hard? Could you build it? What if I got the data for you? No? But you can use it right? They are not perfect, mostly I think because of data cleanliness problems, but they are close enough that I don't travel far from home without one.

More examples. Anyone used a search engine? How about an on-line weather forecast? How about web-sites that predict house-values? Recommendation engines like those used by Amazon and Netflix? All heavy analytic cores wrapped in an easy to consume, highly usable front-end.

These are, I think, among the exceptions in analytic applications - good analytics AND good delivery.

I talked about pseudo-analytics in a recent post: shams with no basis in science wrapped in a User Interface with the hope that nobody asks too many questions about what's under the hood. This is not good analytics.

Unfortunately even good analytic tools get under-used if they have not been made accessible to the poor people that have to use them. Spreadsheet tools probably top the list for unusable analytic applications: unusable that is by anyone except the person that wrote them. Sadly though, I have seen many examples both in reporting and applications where so little effort was put in to User Experience that any good analytics is almost completely obscured.

Building new analytic capability is a highly skilled job. Delivering analytic results in an easy to consume format so that it gets used is also a highly skilled and, frankly, often forgotten step in the process. After all we do build analytic tools so that they get used. Don't we? Sometime I wonder.

Averages work ! (At least for ensemble methods)

After an early start, I was sitting at breakfast downtown enjoying a burrito and an excellent book on "ensemble methods".  (Yes, I do that sometimes... don't judge)

1. 1.

Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions (Synthesis Lectures on Data... by Giovanni Seni, John Elder and Robert Grossman (Feb 24, 2010)

Recommended Reading: The Definitive Guide To Inventory Management

A little over 15 years go now, I was set the task to model how much inventory was needed for all of our, 3000 or so, products at every distribution center.  Prior to this point, inventory targets had been set at aggregate level based off experience and my management felt it was likely we had too much inventory in total and what we did have was probably not where it was most needed. (BTW - they were absolutely right and we were ultimately able to make substantial cuts in inventory while raising service levels).

I came to the project with a math degree, some programming expertise, practical experience simulating production lines, optimizing distribution networks, analyzing investments and with no real idea of how to get the job done.  The books I managed to get my hands on gave you some idea how to use such a system but no real idea how to build it.  They left out all the hard/useful bits I think.  So, I set about to work it out for myself with a lot of simulation models to validate that the outputs made sense.

I still work occasionally in inventory modeling and I'll be teaching some components this fall, so I have been eagerly awaiting this new book : The Definitive Guide to Inventory Management: Principles and Strategies for the Efficient Flow of Inventory across... by CSCMP, Waller, Matthew A. and Esper, Terry L. (Mar 19, 2014)

Data Visualization - are pie-charts evil ?

I'll be speaking next week at the Supply Chain Management Conference at the University of Arkansas on how data-visualization enables action.   

Good visualization is fairly easy, unfortunately, building bad visualizations that are hard to use, easy to misunderstand and that obscure and distort the data you are trying to present is even easier - many analysts can do it without trying to.



In honor of the event, I'm resurrecting a post I created a couple of years ago "Are pie charts evil or just misunderstood".  I wrote this around  the time I was moving away from a trial and error approach  (and 20 years of trial and error effort does get you cleaner visuals) to attempting to understand why some visuals so clearly work better than others.  

It turns out that there are some great frameworks to help in building better visuals.  Join me next week and we'll talk about human graphical perception, chart junk and non-data ink.

Enjoy !

Data Visualization - enabling action

I'll be speaking next week at the Supply Chain Management Research Center Conference at the University of Arkansas on how data-visualization enables action.

The basic premise (and one I firmly believe) is that the hardest part of any analytic project is not defining the problem, doing the analytics or finding the "solution", it's enabling action. Far too many otherwise excellent analytic projects, tools and reports go unused because the results are presented in a way that is somewhere between difficult-to-understand and incomprehensible.

Back to blogging on "Better Business Analytics"

It's been quite a while, just over 12 months in fact since my last blog post.  In that time, I've been hard at work developing analytic applications for the Orchestro DSR.  (Orchestro's off-shelf alerting tool is especially cool and something I am very proud of contributing to).    I enjoyed my time at Orchestro, they're a good team and have big plans, but one key thing I found out about myself is that I prefer working real-life problems to developing software for someone else to have all the fun :-)

So, I'm now back full-time on consulting and I will occasionally blog on topics of interest to me.   Expect to see more soon on:

• Next-generations DSRs (Demand Signal Repositories)
• Retail supply-chain analytics
• Handling (BIG-ish) data for analytics
• The right tools for the job (Predictive Analytics, Business Models, Optimization)
• Some more thoughts on store-clustering
• Inventory modeling at retail (and why it's different, again)
• Order forecasting using POS data
• Further thoughts on SNAP and other ignored demand drivers
• and if there is something you would like to hear more on ... just drop me a line.

Business Analytics - finding the balance between complexity and readability

In this blog I try to present analytic material for a non-analytic audience.  I focus on point of sale and supply chain analytics: it's a complex area and frankly, it's far too easy whether writing for a blog or presenting to a management-team to slip into the same language I would use with an expert.  

So, I was inspired by a recent post on Nathan Yau's excellent blog FlowingData to look at the "readability" of my own posts and apply some simple analytics to the results.

The right tools for (structured) BIG DATA handling (update)

A couple of weeks ago, I ran a somewhat rough benchmark to show just how much faster large database queries can run if you use better tools.

The right tools for (structured) BIG DATA handling  Here's the scenario: you are a business analyst charged with providing reporting and basic analytics on more data than you know how to handle - and you need to do it without the combined resources of your IT department being placed at your disposal.  Sounds familiar?

I looked at the value of upgrading hard-drives (to make sure the CPU is actually busy) and the benefit of using columnar storage which let's the database pull back data in larger chunks and with fewer trips to the hard-drive.  The results were ..staggering.  A combined 4100% increase in processing speed so that I could read and aggregate 10 facts from a base table with over 40 million records on my laptop in just 37 seconds.

At the time I promised an update on a significantly larger data-set to see whether the original results scaled well.  I also wanted to see whether query times scaled well to fewer facts.  Ideally querying against 5 facts should take about 50% of the original 10 fact aggregation queries.

The right tools for (structured) BIG DATA handling

Here's the scenario: you are a business analyst charged with providing reporting and basic analytics on more data than you know how to handle - and you need to do it without the combined resources of your IT department being placed at your disposal.  Sounds familiar?

Let's use Point of Sale data as an example as POS data can easily  generates more data-volume than the ERP system.  The data is simple and easily organized in conventional relational database tables -  you have a number of "facts" (sales-revenue, sales-units, inventory,  etc.) defined by product, store and day going back a few years and then some additional information about products, stores and time stored in master ("dimension") tables,

The problem is that you have thousands of stores, thousands of products and hundreds (if not thousands) of days - this can very quickly feel like "big data".    Use the right tools and my rough benchmarks suggests you can not only handle the data but see a huge increase in speed.

Business Analytics - The Right Tools For The Job

Whether your analytic tool of choice is Excel or R or Access or SQL Server or ... whatever,  if you've worked a reasonable range of analytic problems I will guarantee that at some point you have tried to make your preferred tool do a job it is not intended for or that it is ill-suited for.  The end result is an error-prone, maintenance nightmare and there is a better way.

Business Analytics - The Worst Use of Excel ever ?

Excel is a great tool and I use it a lot.  It's available on almost every business user's desktop and it's highly extensible (with some sensible design) through add-ins and programming but it can't do everything; push it too far and the results can be nasty.  

Here are my nominations for "The Worst Use of Excel ever" awards.

Recommended Reading: Supply Chain Network Design

I've done a lot of  supply chain network design projects and consider myself to be an expert. Had I had this book from the start, I may have got to expert status a lot faster.

With experience in supply-chain and an academic background that includes mathematical-optimization, when the need arose to build supply chain network optimization models I just did it.  Then I learned many, valuable, real-world lessons the hard way- by getting it wrong.

There are a number of books available that cover this area: I have dipped into a few, as needed, and I have not read most of them so I really can't say this is the best book available on the subject.  I can say this is one of the very few analytic books on any subject that I have read cover to cover.  

Better Business Analytics - 2013 New Year's Resolutions

10 resolutions for Better Business Analytics

Firstly - thank you Santa for reading my Christmas list. I love the T-shirt - "Statistics means never having to say you're certain".  With the holiday season coming to a close my thoughts are turning to the New Year and even a certain excitement about getting back to work.  Time for some new year's resolutions !

Better Business Analytics - Christmas list

It's that time of year again: my kids have written, re-written and re-re-written their Christmas lists now so we all hope Santa will read them carefully and take notice.

With just a few days left before the holiday season hits I wanted to do something a little more fun, so I've pulled together a list of things that I think every Business Analyst should want.  Some are free to acquire, just costing your time to learn, others you may wish to ask Santa (or your CFO) to provide.

Better Business Reporting in Excel - XLReportGrids beta released

In my last Blog entry, I talked about reporting in Excel, some of the reasons I choose to make heavy use of  them, and some of the issues that stop me using them even more. (see Better Business Reporting in Excel)

• Pivot-tables can only show you data (although lots of it)
• Pivot-charts show you a chart, but only 1 per pivot-table
• If you want the same report for multiple grouping (e.g months, years, brands or locations) you add these groups to your pivot-table and select them one at a time to print.

XLReportGrids is a free Excel add-in, now available in beta test, that builds grids of reports with multiple copies of a template sized to fit the page.   Templates are just a range of cells in a worksheet that are driven by a pivot-table.  Build templates with: charts, formulas, images, pivot-tables, text boxes, anything that can be added to a worksheet.

Better Business Reporting in Excel

What do you think is the most-used reporting tool in the world?  There are a lot of them available, at times it can be hard to move without running into a new Business Intelligence (BI) tool, but I'm going to hazard a guess with a high degree of confidence that the most used tool is still Excel.

Excel may not be where the data originated from and there may well have been a database involved to crunch numbers and aggregate to the point that the data would fit comfortably in Excel, but I do believe that it's still the most widely used tool for final 'analysis' and presentation of data.

This is not because Excel is the best tool for the job as almost every feature of Excel is handled better by another tool.  Joining separate tables of data with Lookup functions is a very pale shadow of using a properly defined database and the SQL language.  There are substantially better graphics, charting and visualization tools available.  The analysis tools (Correlation, Regression, ANOVA, Linear-programming) are the poor relation to more industrial strength packages.

WARNING: Bad business analytics may be hazardous to your wealth !

You paid handsomely for the software, perhaps for consulting too and have had some bright sparks working on it for months: the results of your analytics project are in and the answer is ... useless without some understanding of how good the models are it's built on.  If the analyst cannot give you detail on how 'good" the model is for its purpose, all results should come with a wealth warning. 

BAD BUSINESS ANALYTICS MAY BE HAZARDOUS TO YOUR WEALTH.

Is the juice worth the squeeze?

I have heard this phrase a lot in recent months in a business context.  It’s so visual, I love it! 

It’s not quite enough though.  It’s pretty simple to understand that every project must be able to pay for itself and deliver a return.  Is the juice worth the squeeze?

It’s also true though that no organization has infinite resources of time or money.  If you have 10 projects that you could do but only enough resources to handle 3, you must prioritize those projects that help you meet your objectives (growth, profitability, market share).  What has the most bang for the buck? 

So with these 2 phrases in mind, it should now be easy…right? (Can you hear the sarcasm)?

Are pie-charts evil or just misunderstood ?

OK - I admit it: when I see a pie-chart in business analytics, my blood pressure rises and, yes, I am apt to tell the analyst exactly what I think of the monstrous, hard to read, waste of space and ink they created.

I am most definitely NOT the first person to suggest that pie-charts are over used and close to useless.  Google 'Pie Charts are Evil' and see for yourself.  This is an area where career analysts tend to agree, yet, in practice, pie-charts are very, very heavily (ab)used.  In the hope that I can influence even a handful of people to create fewer of these eyesores I'm adding my voice to the argument.

Analytic tools "so easy a 10 year-old can use it"

If you search the web you'll find lots of analytic tools to support your business: tools to help with forecasting, inventory optimization, risk analysis, simulation for production lines and warehouses, production scheduling, supply-chain network design, vehicle loading, price-sensitivity modeling and planogram building - and that is very, very far from being an exhaustive list.

Some of these tools are bought as a service that includes expertise to prepare your data, do the modeling work for you and configure the system to meet your needs. These tools will be much more expensive than the 'roll your own' variety and the more frequently that expertise is required, the more you will pay.