Transcending the Current Data Management Paradigm

The future of data management is the elimination of duplicate data.

What exactly does this mean?

Every agency uses various systems internally, all of which store client data.

It’s regularly the case that some client data stored in one system needs to be available in another system, and there may or may not be a clean way to bring this data over in an automated fashion. Even if the data can be brought over initially, what happens if the source data changes? Will the data stored in the second system automatically update, or will a human have to remember to update this data? What if you’re not dealing with two systems, but many?

Unfortunately, the larger the agency, the more systems you probably use, you probably store more client data, and any change to the storage architecture becomes increasingly painful. Also, if your data isn’t automatically updated in systems which have initially consumed your data from another system, larger problems arise such as vast data discrepancies.

Currently, the best solution to this problem is to store all of your data in one system, and connect via API to the rest. When considering all of the long term problems that can arise, along with enormous cost in the form of time involvement to migrate or manage data in multiple systems, I’m sure you can already see the value in this. Of course, the true value of the API is whether or not the consuming systems will receive updates to data when changes are made within the source system. Even if this costs you a pretty penny up front, it’s worth it.

By creating an agile data management system within your organization, you’ll be able to make changes gracefully in the future, and you’ll save your employees a ton of time and tedious work. You might even avoid a few customer service disasters that could occur if you consider the dangers of data discrepancies.

Example: You store customer information that is specific to billing within your invoicing and payment system. Customer data related to deliverables are stored in a CRM. Project milestones, tasks, and notes are stored in a Project Management system. Finally, customer support is handled in a help desk or ticket system.

While this may sound like a lot, it’s really pretty standard, and the scary part is, for many companies this can be a lot worse. There may be certain internal groups tracking particular projects or initiatives within documents that only certain individuals have access to, or there may be a number of additional systems that are more directly related to the deliverables your company produces.

The flip side of this coin is the old adage about putting all of your eggs in one basket. While today this is definitely still a valid argument, I believe the day will soon come that this concern is finally laid to rest, if you’re willing to pony up a bit of a premium cost. Depending on your internal budget, this is more or less already available. The introduction of cloud storage and data redundancy demands this day come to fruition.

Chances are, this all sounds good, but your company just isn’t ready for this change, or doesn’t have the ability to make this change for one reason or another.

Consider improvements you can identify and take action on now, such as those which are easy to see the value of, or very easy to implement. Also, search for data that is heavily duplicated. Chances are, you won’t have the ability to have information in one system automatically update when changed in the source system now, so also search for data that rarely changes.

Search for developers experienced with the API for your most popular system which you could see acting as the main data source. Also, stay on the lookout for services such as Zapier which is taking huge strides in this direction.

Applying this solution to WordPress is the true nature of this idea.

We’ve all seen the WordPress plugins which add a ticket system, or a CRM, or a project management system to WordPress. The problem is that the level of support for these plugins is no comparison to competing applications outside of WordPress, at least for the time being.

The company who realizes the value of this solution, who can create and support the framework to run a culmination of these services within WordPress, and do so in a way that can truly rival awesome products like Zendesk, Teamwork, etc. will be wildly successful and their product will transcend the current data management paradigm.

2 thoughts on “Transcending the Current Data Management Paradigm”

  1. Great post Bob! You have really given our company insight into this subject over the years and improved our product because of it. What are your thoughts about the new REST API developments in WordPress and how those features can continue the elimination of duplicate data? This is a subject you and I have spoken about nth fold, but I personally believe it could revolutionize many markets.

    Also, you should use Droid Serif as your body font for this theme, I have a hard time with sans-serif white on black. };]

    1. Thanks for the feedback, I’ve updated the font as recommended.

      The second version of the REST API looks very promising, however, one thing I haven’t seen mentioned in their documentation is built in support for settings, such as data stored within the Customize UI or the WordPress options table.

      My initial thought would be to map out a schema for data to be stored in endpoints. By extending the REST API in this manner you could create an individual endpoint for each site on a network which could contain base information which isn’t supported out of the box by the REST API, such as data stored in the options table or custom settings such as data stored in the Customize UI. Extending this further, you could also create an individual endpoint for each site for various modular components, which would tie into a network level endpoint connected to the site level endpoints for that modular component.

      If the visualization I have in mind is accurate, this means you could easily store site specific data at the site level, keep it modularized, and still access all of the data at the network level for overall monitoring and upkeep, with very little data duplication.

      Modularization will be the key to keeping the code clean, extendable, and reduce the complexity of the data sharing.

      For me, the core question becomes choosing the appropriate place to store the source data that will be shared amongst modular endpoints.

      The agility for data management this approach brings is extremely powerful since all of the data would be accessible by WordPress. This makes mass data processing functions personal and extendable at will, rather than dealing with limitations of a third party API.

      Last, if the need arises to migrate data to another system at some point, if the new system has any type of import feature, you would be able to build your own data set to import which could be created to be compatible with the new system, and this could all be done programmatically.

      I could see this being the case in which a new tool enters a given industry that you aren’t comfortable with rolling into your system until you learn about the tool a bit, see it in action, and document it’s pros and cons, limitations or flaws, as it pertains to your organization.

      After using the tool in it’s native environment for a while, if you conclude it’d be best to make it part of your core data and create your own in house version of the tool, you could then export your data from the existing tool, create your own custom import tool to map the data appropriately into your new data schema and endpoints. In this way, you would bring your work with you, and eliminating any duplicate data could be one of the steps in the migration process.

Leave a Reply

Your email address will not be published. Required fields are marked *