Working with 'big data'

[Tdarcos]

IT World Magazine has a white paper on 'big data', how they expect the amounts of data used even now to explode by a factor of ten-fold. New sources of data such as blogs, video, social media, logging data, photos, etc. are going to increase the amount of data being used.

Unlike most of these white papers that these magazines put out, it's not being used to sell some vendor's solution (the last page does carry an ad for the sponsor), but explains how some companies have used their analysis of all the new data being used to figure ways to be more profitable or to provide reasons for customers to choose them, often over less expensive competitors.

One of the points it mentions is how in-memory processing is going to be a big thing. I know that from something I heard about and saw at 2011's FOSE exposition, how one company had sold huge amounts of SSD hard drives in the 64 GB size (I think the largest you could get then) to Comcast, which is using them to store video files for use in video on demand. This is a big win-win: storing video in solid state memory has no moving parts, is much faster than a (magnetic) hard drive, allows for a huge amount of read requests, and is perfect for customers who can randomly access a streamed video file as they can show the same video program (a TV show, a movie, a short) to any number of customers from any point in the file.

Another point it makes: "Responsiveness—being agile enough to respond to changing customer demands—is more profitable than efficiency" I remember reading about a local paint company, Duron, that is able to get a premium price for its paint even though having ruthless competitors such as Home Depot and Lowes, and it can because it will guarantee delivery time on orders. A contractor who runs out of paint may lose considerably more than the cost of paint if they have to send someone out to get it and the job is left undone with people sitting, getting paid and not working, because they're out of a critical resource.

If you have to work with large amounts of data it's definitely worth reading, it's set up as a wide page (like a magazine article or Powerpoint presentation, although not that badly dumbed down) to show in a web browser, and it's only 15 pages. Definitely worth reading; it may give you some ideas about things you might want to do relative to the data your organization generates.

http://resources.idgenterprise.com/orig ... g_Data.pdf

[CO]

Bravo to Tdarcos for a great post, very informative and thoughtful. The article you linked to actually will help me in my current job because now I have a cool new industry buzzword to describe what I sell: DaaS (Data as a Service) . Thanks commander!

[RetroRomper]

While researching "cloud" backup systems, I've been exposed to the term "data dedupe" (which sounds coming from the mouth of a sales exec as "data dedoop.") For a while I didn't understand the exact term but after being exposed to it seven or eight times, realized its merely a buzz word for a compression method (and when coupled with a service that is paid for by the gigabyte, becomes remarkably relevant).

So an entire industry is using a phrase more asinine than "the cloud" to describe a phenomenon I've been exposed to all of my life.

[Tdarcos]

While researching "cloud" backup systems, I've been exposed to the term "data dedupe" (which sounds coming from the mouth of a sales exec as "data dedoop.")

It means that you try to reduce redundant data in order to reduce the possibility of erroneous data. If their address is listed in five places, if at least one is right, you potentially could have four wrong. Maybe 5 wrong, but if you only have one address record, you only have one item and you don't have to guess which one is right.

So, if someone is in a set of files in multiple places, you want to have one address record for them, and anywhere else you should point back to their address, not duplicate it. And possibly have their name in one place, potentially with links to it.

It's a standard practice in data normalization, where you try not to repeat data in multiple places.

[Flack]

Here's a great article I recently read about an industry that uses more bandwidth and storage than almost any other: porn.

http://www.extremetech.com/computing/12 ... porn-sites

[Tdarcos]

Here's a great article I recently read about an industry that uses more bandwidth and storage than almost any other: porn.

Most people are unaware that almost everything related to Internet commerce was developed as a result of selling porn. HTTPS, credit card processing for web sites, password protection of web sites, etc.