No Matter How Great - and it IS Great - This is One Thing Big Data Cannot Do

"The project would not have been started if the truth had been told about the cost and timescale." - Unknown, quote from  http://www.famous-quotes.net/Topic.aspx?Project_Management

To treat your time as preciously as possible, please read the following statements.  If you have NEVER said, heard or overheard (there is a difference) any of these, please feel free to stop reading now and go over to http://www.smartertechnology.com or http://www.internetevolution.com and do a little learning about big data.  Now for the statements:

- Officially the manager doesn't want any administrative costs entered.  We know there are always administrative costs, so just fold them into whatever you spent most of your time on.

- Make sure you spend more than $15 on supper, otherwise Accounting will reduce our per diem.

- We know they found a problem in testing, but the release date is tomorrow.  Ship it, and make sure anyone who gets the first build gets the fix.

- We have monthly inventory tomorrow, so load up all the machines, even the trimmers, and make sure the scrap goes on the trimmers.  All corporate cares about is raw materials, scrap and finished goods in the warehouse.  The more we have in process the less we have to explain.

- He's out for his three hour daily lunch.  He needs to be back by 2:00 though so he can make his 4:00 tee time.

Still with me?  I thought you would be. All but one of those I've heard firsthand.  In fact I was once called on the carpet for eating fast food instead of something more substantial for lunch, but that's another story, from a company that has since gone out of business. Follow me for a bit here.

Big data is a revolution in the way that things are monitored, analyzed, predicted and operated.  It really is fascinating stuff, and I've seen demonstrations of its power in action, grabbing input from Twitter, Facebook and a few other sites and using comments and such to determine the relative quality of a laptop, for example.  I sometimes refer to it as 'butterfly effect' computing, because its single-minded pursuit is the purchase of a pack of gum that foretells the results of a recall election.  That isn't written with any humorous intent - it truly is the last great frontier to cross in order to be able to put a blueprint to the seemingly patternless, and often hinges on the correlation between strange things.  It is a game of digital poker, learning the tells and the styles of everything and using that knowledge to position your hand to win.

BUT...

There is a factor which big data really cannot evaluate, at least not yet.  It is the ethics of what we do, and the connection between what we do and what the data say we do.  For example, I used to work for a boss who was constantly in the pursuit of receipts.  He would gladly accept dining receipts from any area restaurant, and  was very diligent in the filing of his expense reports.  You see, without going into a lot of the details, he spent a  lot of his money in $1 increments, and was not really able to request a receipt for the bills, nor would the receipts be accepted by Accounting.  So, he turned in his 'business meals' on his expense reports, supported by the receipts, and I would assume recouped his spendings.

The system said there must be receipts, valid receipts were submitted, so the system was satisfied.  The same system was not satisfied with allowing tips, though they are customary when you are actually dining out, so those had to be rolled in differently.  The system also didn't keep track of when a restaurant was open, so a receipt claimed as lunch from a restaurant open only from 5:00 p.m. onward was fine. 

You could find examples of driving logs that are all written in the same ink, though they span a year. (Truthfully, other than the one in the decorative desk set, has anyone anywhere ever held onto the same pen for more than a month?)  You could find (and I did find at the same place that was angry about a trip to Burger King) the same parts being rejected at an 80%+ scrap rate by one division, and only at a 20%+ rate by another division, the personal relationships affecting business decisions, making things work when they shouldn't, and burying the bodies in jointly-held cemeteries.

It is actually time for a wholesale change in the way that business is conducted.  Big data can look at the things that are reported and make the analysis based on what is there, but big data cannot find the actual truth in all matters.  It cannot tell you the actual cost of a project, the actual breakdown of expenses, or the actual aggregate series of events in a project.  The reason is a confectioner's treat.

In business, there is a long-standing thing called the fudge factor, whereby it is naturally assumed that there needs to be a margin of error, so you overbudget, overestimate time, overestimate cost (unless you are in an ERP project - to my knowledge there has NEVER been a cost OR time overestimation of one of those) while you also undercommit and overdeliver.  When you have a supplier and a consumer, each of them has their own fudge factor, and the two fudge factors can wind up creating a surplus of fudge.  That surplus enters the datastream, which then is used to figure out other things.

As with all software issues, the problem originates in the carbon-based units.  No matter how good a tool is, it still relies on the human input to determine how it is working.  It also must work within a system that was designed and implemented by humans many years ago.  To be blunt - we have never in the history of the human race been able to have such potential accuracy, yet we cling to the inaccuracy of the fable because we perceive the inaccuracy to be a necessary part of our process, and then we pass the disease on to our data.  Consider that - we have the purest form of power there is, data and the tools to analyze it - yet we would rather infect it with our inaccuracies than let it tell us where we can truly have an advantage.

I am a big believer in the concept of never bringing a problem without a solution, but in this case I can't do that.  Here's why.  The human factors reign supreme.  As an illustration, one problem I have blogged about in the past is the meaningless metric.  If someone processes 100 files a day, and you bring in a system that lets them process 200 files a day, in order for it to be a win there has to be 1) actually an additional 100 folders for them to process to necessitate the need, and/or 2) another processor position that you then cut in order to save and have 200 folders a day represent what it was being presented as being.  No salesman will EVER tell you in an open meeting that you can reduce headcount with a product, but do the math. The whole metric is meaningless unless you have the demand that the new system would fill and the political will to cut people until the savings are realized...at least as far as you think they are.

Which means that you make the assumption that people can work with 100% efficiency, and cut until you think they are at 100%, which then leads to sick time and if additional needs enter into the realm of what is being done you have training expenses and down time and, and, and... you trade potential for short-term gains, mostly on paper.  Do you tell the full-timer they are going to part-time?  Maybe, but they also may leave and you will then have to train their replacement.  I've been gone almost a year from a previous employer and they occasionally still send me an e-mail, asking about this or that.  Do all former employees answer the questions you didn't know needed answers before they left?  I do, but I'm likely in the minority.  So even if the new system is put in place, the headcount doesn't change, and possibly the need for efficiency isn't there either because of low demand.  Big data tools could tell you, but you have to tell them the right thing first.

You can say that the per diem will never adjust downward as a matter of policy, and that will get rid of the whole lunch and dinner expense fiasco, but then the human factor enters in and someone who always has to entertain clients complains about someone else who never has to and thus eats fast food and pockets the rest of the per diem (because the fix of just paying the per diem regardless was the first solution they tried).

The only fix I can suggest is that we reframe how business is done so that instead of garbage numbers we are feeding our systems the truthful story.  An executive who states that they don't want to see any administrative overhead within a project is just wrong.  I don't mean to offend with this statement, though I stand behind its logic: there cannot be a project plan without time being spent setting up meetings, running interference, adjusting communications, explaining and justifying to the C-level, etc.  Big data tools can help to streamline and shape that time to find better ways of doing things, but ONLY IF IT IS POSSIBLE TO KNOW ABOUT THEM!  If you say the cost is $150, the data do not know that you spent $127 on the component and the rest on the tip for the delivery man, they only know the check was for $150.

In the same vein, accounting policies and procedures can cause more data headaches than they solve.  The policies have evolved from the receipt-based concept to more of the per diem concept, but in the end they are not informative or accurate enough to allow a true handle on the costs.  If a company is headquartered in Kansas City, but they have a Manhattan office, my guess would be that the expenses of the Manhattan office would be far greater than those of the Kansas City office.  Do they make different amounts the standard per diem depending on where the employee is located, or do they find a mutually agreeable number for all involved, and the guys in Kansas City brown bag their lunches and pocket some extra money?  This has Human Resources hassle written all over it, so the business response is to equalize everything and pad the number such that the complaining reduces to an acceptable amount.

For big data tools to truly help, a base level of honesty needs to pervade the corporate culture.  For them to truly predict costs, they must have an accurate baseline.  For them to determine market trends, they must truly have a grasp of raw materials versus in-process materials, finished stock versus returns.  For almost every area in which a creative accounting method has been developed, there is a potential peril for using big data to analyze.

I suppose I can say there is a fix, and it is a two-part fix.  Part One is to report things as accurately as possible.  That only makes sense.  The tools involved in any project need accurate inputs so that they can give accurate outputs.  Can a system you lie to tell you the truth at a later time?

Part Two?  That is the tough part.  Since there is now, and always has been , such a cloak over the accurate data, it will take time for things to normalize.  The second part of the solution is to be okay with what you see in the analysis.  So many things in business - the fudge factor, the inflated receipts, the hiding of administrative expenses, the re-classification of raw materials - lead to false assumptions and bad strategy.  The second part of the solution is just being okay with the results.  Over time the analysis will grow better and more accurate, but in the beginning there is a lot of fudging to root out.  It is a concept that hearkens back to our agricultural heritage - first you muck out the barn thoroughly, then after the water dries you see how much room there actually is and where you should build your new stalls. Without the mucking, there is no accurate plan for building.  That needs to not only be understood, but demonstrably accepted by the management.  Then you can bring in the big guns of data and blow a hole through your paradigms to true efficiency and profitability.  Make sure you keep the receipts for the ammo, though...