Better Business Analytics

The Next-Generation DSR

CPGs have had access to Point of Sale (POS) data now for many years and many of them use a Demand Signal Repository (DSR) to gather, clean and report on this data. (Actually most of them use a number of DSR's and even when they do have just one, still can't handle truly cross-retailer analytics).

I've been involved with a number of these systems as a software-buyer, a system-administrator, consultant and most recently, leading the analytic development at Orchestro.

There are some excellent tools available and, in their current form, they can help you drive both additional revenue and reduced costs when used well. However, in my experience many of these tools have been sold in under the guise of "saving time" through reporting automation. That's valuable, but it's not "finding a new sales opportunity" valuable.

I think we are still in the infancy of DSR development: systems are operating at the limits of the technology they were built on and necessary trade-offs mean that being good at one thing (e.g. speed) makes it more challenging to be good at others (e.g. analytics).

The next generation of DSR can be dramatically more effective. In particular, it will be:

Back to blogging on "Better Business Analytics"

It's been quite a while, just over 12 months in fact since my last blog post. In that time, I've been hard at work developing analytic applications for the Orchestro DSR. (Orchestro's off-shelf alerting tool is especially cool and something I am very proud of contributing to). I enjoyed my time at Orchestro, they're a good team and have big plans, but one key thing I found out about myself is that I prefer working real-life problems to developing software for someone else to have all the fun :-)

So, I'm now back full-time on consulting and I will occasionally blog on topics of interest to me. Expect to see more soon on:

Next-generations DSRs (Demand Signal Repositories)
Retail supply-chain analytics
Handling (BIG-ish) data for analytics
The right tools for the job (Predictive Analytics, Business Models, Optimization)
Some more thoughts on store-clustering
Inventory modeling at retail (and why it's different, again)
Order forecasting using POS data
Further thoughts on SNAP and other ignored demand drivers
and if there is something you would like to hear more on ... just drop me a line.

Business Analytics - finding the balance between complexity and readability

In this blog I try to present analytic material for a non-analytic audience. I focus on point of sale and supply chain analytics: it's a complex area and frankly, it's far too easy whether writing for a blog or presenting to a management-team to slip into the same language I would use with an expert.

So, I was inspired by a recent post on Nathan Yau's excellent blog FlowingData to look at the "readability" of my own posts and apply some simple analytics to the results.

The right tools for (structured) BIG DATA handling (update)

A couple of weeks ago, I ran a somewhat rough benchmark to show just how much faster large database queries can run if you use better tools.

The right tools for (structured) BIG DATA handling Here's the scenario: you are a business analyst charged with providing reporting and basic analytics on more data than you know how to handle - and you need to do it without the combined resources of your IT department being placed at your disposal. Sounds familiar?

I looked at the value of upgrading hard-drives (to make sure the CPU is actually busy) and the benefit of using columnar storage which let's the database pull back data in larger chunks and with fewer trips to the hard-drive. The results were ..staggering. A combined 4100% increase in processing speed so that I could read and aggregate 10 facts from a base table with over 40 million records on my laptop in just 37 seconds.

At the time I promised an update on a significantly larger data-set to see whether the original results scaled well. I also wanted to see whether query times scaled well to fewer facts. Ideally querying against 5 facts should take about 50% of the original 10 fact aggregation queries.

The right tools for (structured) BIG DATA handling

Here's the scenario: you are a business analyst charged with providing reporting and basic analytics on more data than you know how to handle - and you need to do it without the combined resources of your IT department being placed at your disposal. Sounds familiar?

Let's use Point of Sale data as an example as POS data can easily generates more data-volume than the ERP system. The data is simple and easily organized in conventional relational database tables - you have a number of "facts" (sales-revenue, sales-units, inventory, etc.) defined by product, store and day going back a few years and then some additional information about products, stores and time stored in master ("dimension") tables,

The problem is that you have thousands of stores, thousands of products and hundreds (if not thousands) of days - this can very quickly feel like "big data". Use the right tools and my rough benchmarks suggests you can not only handle the data but see a huge increase in speed.

Business Analytics - The Right Tools For The Job

Whether your analytic tool of choice is Excel or R or Access or SQL Server or ... whatever, if you've worked a reasonable range of analytic problems I will guarantee that at some point you have tried to make your preferred tool do a job it is not intended for or that it is ill-suited for. The end result is an error-prone, maintenance nightmare and there is a better way.