/9918578/Banner

/9918578/Banner_2

/9918578/Mid

section

articleId

product

orgType

parrefer

accountID

Consulting

Banking

Accounting

Fintech

Internships

Rankings

Internship Rankings

Intern Resource Center

Explore Internships

Vault Law Home

Practice Area Resource Centers

Regional Resource Centers

Summer Program Resource Center

Vault Law Culture & Inclusion Survey

Vault Law

Guides

Events

Financial Services

Technology

Grad School

Work Relationships

Productivity

Diversity, Equity & Inclusion

Social Impact

Remote Work

Negotiating

Changing Jobs

Workplace Issues

Blog

Resume Samples

Cover Letter Samples

Interview Tips & Questions

A Day in the Life

International

Career Readiness

Explore Professions

Research Industries

Search Companies

Career Advice

Employers

Schools

“I’ve seen that before”.&lt;p&gt;This sentence, in essence, describes a technology known as de-duplication. In the previous post we described the importance of “dedupe”. The amount of world-wide digital data being generated daily is enormous, and dedupe is a strategic technology used to reduce the amount of information that actually gets stored.&lt;/p&gt;&lt;p&gt;If you don’t know anything about this technology, you are at a disadvantage as an interview candidate. If you already have a high-tech job, you need to add dedupe to your knowledge portfolio.&lt;/p&gt;&lt;p&gt;At a very high-level you can think of dedupe as a “pattern recognition” technique. Consider the following diagram, which depicts a data set being sent through a dedupe service.&lt;/p&gt;&lt;p&gt;&lt;img src=&quot;https://media2.vault.com/39413c13-824e-4f0b-8987-5360da2d4c49/00000150.png&quot; alt=&quot;Dedup Recognizes and Reduces Identical Data&quot; border=&quot;0&quot; class=&quot;embeddedObject&quot; /&gt;&lt;/p&gt;&lt;p&gt;Consider our foundational diagram that depicts an application sending information to a persistent storage device. A Dedup Service scans every bit of information that flows from the application and looks for identical bit patterns. When it recognizes identical patterns the service “dedupes” the data and does not store it persistently. In the diagram above, the bit pattern “0101” occurs four times, but it is only stored once. This example results in a 25% capacity savings on disk. For huge data sets, a 25% capacity savings is significant.&lt;/p&gt;&lt;p&gt;This example also makes it clear that backup applications (which often back up the same data over and over again) are a great use case for this technology.&lt;/p&gt;&lt;p&gt;The dedup service can be implemented in different locations, as depicted in the following diagram.&lt;/p&gt;&lt;p&gt;&lt;img src=&quot;https://media2.vault.com/96803a89-5517-47a2-a952-8dc0b7badd96/00000151.png&quot; alt=&quot;Dedup in Sever versus Storage&quot; border=&quot;0&quot; class=&quot;embeddedObject&quot; /&gt;&lt;/p&gt;&lt;p&gt;Why implement dedup in one location versus another? As seen on the left, running dedup in the server results in less data being sent to a storage device, but more CPU time being spent in the server (remember that dedup must spend CPU time examining every bit of information). On the right, the server spends no time doing dedup but must send every bit of information over to the storage device. Customers often choose one dedup method over another based on their business requirements.&lt;/p&gt;&lt;p&gt;I encourage high-tech job seekers to learn more about de-duplication (especially what’s going on under the covers). For a real-world example of a dedup service running inside of an application server, I recommend reading about EMC’s Avamar product. For dedup running inside of a disk array, study EMC’s Data Domain product.&lt;/p&gt;&lt;p&gt;Steve&lt;br /&gt;http://stevetodd.typepad.com&lt;br /&gt;Twitter: @SteveTodd&lt;br /&gt;EMC Intrapreneur&lt;/p&gt;

Steve Todd

Article

AI Predictions for 2026: What Job Seekers Should Expect

AI in 2025: The Biggest Shifts, Industry Disruption, and What It Means for Your Career

Our Best AI Career Advice of 2025

Can You Have a Career in the Modern Workplace Without a Smartphone?

The Library