Examine individual changes

''''MapReduce''' is a [[programming model]] and an associated implementation for processing and generating large data sets with a [[Parallel computing|parallel]], [[distributed computing|distributed]] algorithm on a [[Cluster (computing)|cluster]].<ref>[http://news.cnet.com/8301-10784_3-9955184-7.html Google spotlights data center inner workings | Tech news blog - CNET News.com<!-- Bot generated title -->]</ref><ref>[http://static.googleusercontent.com/media/research.google.com/es/us/archive/mapreduce-osdi04.pdf MapReduce: Simplified Data Processing on Large Clusters]</ref> Conceptually similar approaches have been very well known since 1995 with the [[Message Passing Interface]] <ref>http://www.mcs.anl.gov/research/projects/mpi/mpi-standard/mpi-report-2.0/mpi2-report.htm MPI 2 standard</ref> standard having reduce <ref>[http://mpitutorial.com/tutorials/mpi-reduce-and-allreduce/ Tutorial on MPI Reduce and all reduce]</ref> and scatter operations.<ref>[http://mpitutorial.com/tutorials/performing-parallel-rank-with-mpi/ MPI tutorial Scatter and Gather]</ref> A MapReduce program is composed of a [[Map (parallel pattern)|'''Map()''']] [[procedure (computing)|procedure]] (method) that performs filtering and sorting (such as sorting students by first name into queues, one queue for each name) and a '''Reduce()''' method that performs a summary operation (such as counting the number of students in each queue, yielding name frequencies). The "MapReduce System" (also called "infrastructure" or "framework") orchestrates the processing by [[Marshalling (computer science)|marshalling]] the distributed servers, running the various tasks in parallel, managing all communications and data transfers between the various parts of the system, and providing for [[Redundancy (engineering)|redundancy]] and [[Fault-tolerant computer system|fault tolerance]]. The model is inspired by the [[map (higher-order function)|map]] and [[fold (higher-order function)|reduce]] functions commonly used in [[functional programming]],<ref name="map">"Our abstraction is inspired by the map and reduce primitives present in Lisp and many other functional languages." -[http://research.google.com/archive/mapreduce.html "MapReduce: Simplified Data Processing on Large Clusters"], by Jeffrey Dean and Sanjay Ghemawat; from [[Google Research]]</ref> although their purpose in the MapReduce framework is not the same as in their original forms.<ref>{{Cite journal | doi = 10.1016/j.scico.2007.07.001| title = Google's Map ''Reduce'' programming model — Revisited| journal = Science of Computer Programming| volume = 70| pages = 1| year = 2008| last1 = Lämmel | first1 = R. }}</ref> The key contributions of the MapReduce framework are not the actual map and reduce functions, but the scalability and fault-tolerance achieved for a variety of applications by optimizing the execution engine once. As such, a [[single-threaded]] implementation of MapReduce will usually not be faster than a traditional (non-MapReduce) implementation, any gains are usually only seen with [[multi-threaded]] implementations.<ref name=stackoverflow>{{cite web | url = https://stackoverflow.com/questions/3947889/mongodb-terrible-mapreduce-performance | title = MongoDB: Terrible MapReduce Performance | publisher = Stack Overflow | date = October 16, 2010 | quote = The MapReduce implementation in MongoDB has little to do with map reduce apparently. Because for all I read, it is single-threaded, while map-reduce is meant to be used highly parallel on a cluster. ... MongoDB MapReduce is single threaded on a single server... }}</ref> The use of this model is beneficial only when the optimized distributed shuffle operation (which reduces network communication cost) and fault tolerance features of the MapReduce framework come into play. Optimizing the communication cost is essential to a good MapReduce algorithm.<ref name="ullman" /> MapReduce [[library (software)|libraries]] have been written in many programming languages, with different levels of optimization. A popular [[open-source software|open-source]] implementation that has support for distributed shuffles is part of [[Apache Hadoop]]. The name MapReduce originally referred to the proprietary [[Google]] technology, but has since been [[generic trademark|genericized]]. MapReduce as a big data processing model is considered dead by many domain experts,<ref>{{cite web|url=http://www.datacenterknowledge.com/archives/2014/06/25/google-dumps-mapreduce-favor-new-hyper-scale-analytics-system/ | title=Google Dumps MapReduce in Favor of New Hyper-Scale Analytics System |last1=Sverdlik |first1=Yevgeniy |date=2014-06-25 |website=Data Center Knowledge |access-date=2015-10-25 |quote="We don't really use MapReduce anymore" [Urs Hölzle, senior vice president of technical infrastructure at Google]}}</ref> as development has moved on to more capable and less disk-oriented mechanisms that incorporate full map and reduce capabilities.<ref>{{cite web |url=https://gigaom.com/2014/03/27/apache-mahout-hadoops-original-machine-learning-project-is-moving-on-from-mapreduce/ |title=Apache Mahout, Hadoop’s original machine learning project, is moving on from MapReduce |last1=Harris |first1=Derrick |date=2014-03-27 |website=[[Gigaom]] |access-date=2015-09-24 |quote=Apache Mahout [...] is joining the exodus away from MapReduce.}}</ref> ==Overview== MapReduce is a framework for processing [[Parallel computing|parallelizable]] problems across huge datasets using a large number of computers (nodes), collectively referred to as a [[Computer cluster|cluster]] (if all nodes are on the same local network and use similar hardware) or a [[Grid Computing|grid]] (if the nodes are shared across geographically and administratively distributed systems, and use more heterogenous hardware). Processing can occur on data stored either in a [[filesystem]] (unstructured) or in a [[database]] (structured). MapReduce can take advantage of locality of data, processing it on or near the storage assets in order to reduce the distance over which it must be transmitted. * '''"Map" step:''' Each worker node applies the "map()" function to the local data, and writes the output to a temporary storage. A master node orchestrates that for redundant copies of input data, only one is processed. * '''"Shuffle" step:''' Worker nodes redistribute data based on the output keys (produced by the "map()" function), such that all data belonging to one key is located on the same worker node. * '''"Reduce" step:''' Worker nodes now process each group of output data, per key, in parallel. MapReduce allows for distributed processing of the map and reduction operations. Provided that each mapping operation is independent of the others, all maps can be performed in parallel&nbsp;&ndash; though in practice this is limited by the number of independent data sources and/or the number of CPUs near each source. Similarly, a set of 'reducers' can perform the reduction phase, provided that all outputs of the map operation that share the same key are presented to the same reducer at the same time, or that the reduction function is [[Associative property|associative]]. While this process can often appear inefficient compared to algorithms that are more sequential, MapReduce can be applied to significantly larger datasets than "commodity" servers can handle&nbsp;&ndash; a large [[server farm]] can use MapReduce to sort a [[petabyte]] of data in only a few hours.<ref>{{cite web|last=Czajkowski|first=Grzegorz,|title=Sorting Petabytes with MapReduce - The Next Episode|url=http://googleresearch.blogspot.com/2011/09/sorting-petabytes-with-mapreduce-next.html|publisher=Google|accessdate=7 April 2014|author2=Marián Dvorský |author3=Jerry Zhao |author4=Michael Conley |archivedate=7 September 2011}}</ref> The parallelism also offers some possibility of recovering from partial failure of servers or storage during the operation: if one mapper or reducer fails, the work can be rescheduled&nbsp;&ndash; assuming the input data is still available. Another way to look at MapReduce is as a 5-step parallel and distributed computation: # '''Prepare the Map() input''' – the "MapReduce system" designates Map processors, assigns the input key value ''K1'' that each processor would work on, and provides that processor with all the input data associated with that key value. # '''Run the user-provided Map() code''' – Map() is run exactly once for each ''K1'' key value, generating output organized by key values ''K2''. # '''"Shuffle" the Map output to the Reduce processors''' – the MapReduce system designates Reduce processors, assigns the ''K2'' key value each processor should work on, and provides that processor with all the Map-generated data associated with that key value. # '''Run the user-provided Reduce() code''' – Reduce() is run exactly once for each ''K2'' key value produced by the Map step. # '''Produce the final output''' – the MapReduce system collects all the Reduce output, and sorts it by ''K2'' to produce the final outcome. These five steps can be logically thought of as running in sequence – each step starts only after the previous step is completed – although in practice they can be interleaved as long as the final result is not affected. In many situations, the input data might already be distributed ([[Shard (database architecture)|"sharded"]]) among many different servers, in which case step 1 could sometimes be greatly simplified by assigning Map servers that would process the locally present input data. Similarly, step 3 could sometimes be sped up by assigning Reduce processors that are as close as possible to the Map-generated data they need to process. ==Logical view== The ''Map'' and ''Reduce'' functions of ''MapReduce'' are both defined with respect to data structured in (key, value) pairs. ''Map'' takes one pair of data with a type in one [[data domain]], and returns a list of pairs in a different domain: <code>Map(k1,v1)</code> → <code>list(k2,v2)</code> The ''Map'' function is applied in parallel to every pair in the input dataset. This produces a list of pairs for each call. After that, the MapReduce framework collects all pairs with the same key from all lists and groups them together, creating one group for each key. The ''Reduce'' function is then applied in parallel to each group, which in turn produces a collection of values in the same domain: <code>Reduce(k2, list (v2))</code> → <code>list(v3)</code> Each ''Reduce'' call typically produces either one value v3 or an empty return, though one call is allowed to return more than one value. The returns of all calls are collected as the desired result list. Thus the MapReduce framework transforms a list of (key, value) pairs into a list of values. This behavior is different from the typical functional programming map and reduce combination, which accepts a list of arbitrary values and returns one single value that combines ''all'' the values returned by map. It is [[Necessity and sufficiency|necessary but not sufficient]] to have implementations of the map and reduce abstractions in order to implement MapReduce. Distributed implementations of MapReduce require a means of connecting the processes performing the Map and Reduce phases. This may be a [[distributed file system]]. Other options are possible, such as direct streaming from mappers to reducers, or for the mapping processors to serve up their results to reducers that query them. ===Examples=== The prototypical MapReduce example counts the appearance of each word in a set of documents:<ref>[http://research.google.com/archive/mapreduce-osdi04-slides/index-auto-0004.html Example: Count word occurrences]. Research.google.com. Retrieved on 2013-09-18.</ref> '''function''' <u>map</u>(String name, String document): ''// name: document name'' ''// document: document contents'' '''for each''' word w '''in''' document: emit (w, 1) '''function''' <u>reduce</u>(String word, Iterator partialCounts): ''// word: a word'' ''// partialCounts: a list of aggregated partial counts'' sum = 0 '''for each''' pc '''in''' partialCounts: sum += pc emit (word, sum) Here, each document is split into words, and each word is counted by the ''map'' function, using the word as the result key. The framework puts together all the pairs with the same key and feeds them to the same call to ''reduce''. Thus, this function just needs to sum all of its input values to find the total appearances of that word. As another example, imagine that for a database of 1.1 billion people, one would like to compute the average number of social contacts a person has according to age. In [[SQL]], such a query could be expressed as: <source lang="sql"> SELECT age, AVG(contacts) FROM social.person GROUP BY age ORDER BY age </source> Using MapReduce, the <tt>K1</tt> key values could be the integers 1 through 1100, each representing a batch of 1 million records, the <tt>K2</tt> key value could be a person’s age in years, and this computation could be achieved using the following functions: '''function''' Map '''is''' '''input:''' '''integer''' K1 between 1 and 1100, representing a batch of 1 million social.person records '''for each''' social.person record in the K1 batch '''do''' '''let''' Y be the person's age '''let''' N be the number of contacts the person has '''produce one output record''' (Y,(N,1)) '''repeat''' '''end function''' '''function''' Reduce '''is''' '''input:''' age (in years) Y '''for each''' input record (Y,(N,C)) '''do''' Accumulate in S the sum of N*C Accumulate in C_new the sum of C '''repeat''' '''let''' A be S/C_new '''produce one output record''' (Y,(A,C_new)) '''end function''' The MapReduce System would line up the 1100 Map processors, and would provide each with its corresponding 1 million input records. The Map step would produce 1.1 billion <tt>(Y,(N,1))</tt> records, with <tt>Y</tt> values ranging between, say, 8 and 103. The MapReduce System would then line up the 96 Reduce processors by performing shuffling operation of the key/value pairs due to the fact that we need average per age, and provide each with its millions of corresponding input records. The Reduce step would result in the much reduced set of only 96 output records <tt>(Y,A)</tt>, which would be put in the final result file, sorted by <tt>Y</tt>. The count info in the record is important if the processing is reduced more than one time. If we did not add the count of the records, the computed average would be wrong, for example: ''-- map output #1: age, quantity of contacts'' 10, 9 10, 9 10, 9 ''-- map output #2: age, quantity of contacts'' 10, 9 10, 9 ''-- map output #3: age, quantity of contacts'' 10, 10 If we reduce files <tt>#1</tt> and <tt>#2</tt>, we will have a new file with an average of 9 contacts for a 10-year-old person ((9+9+9+9+9)/5): ''-- reduce step #1: age, average of contacts'' 10, 9 If we reduce it with file <tt>#3</tt>, we lose the count of how many records we've already seen, so we end up with an average of 9.5 contacts for a 10-year-old person ((9+10)/2), which is wrong. The correct answer is 9.1<span style="text-decoration: overline;">66</span> = 55 / 6 = (9*3+9*2+10*1)/(3+2+1). ==Dataflow== The frozen part of the MapReduce framework is a large distributed sort. The hot spots, which the application defines, are: * an ''input reader'' * a ''Map'' function * a ''partition'' function * a ''compare'' function * a ''Reduce'' function * an ''output writer'' ===Input reader=== The ''input reader'' divides the input into appropriate size 'splits' (in practice typically 64&nbsp;MB to 128&nbsp;MB) and the framework assigns one split to each ''Map'' function. The ''input reader'' reads data from stable storage (typically a [[distributed file system]]) and generates key/value pairs. A common example will read a directory full of text files and return each line as a record. ===Map function=== The ''Map'' function takes a series of key/value pairs, processes each, and generates zero or more output key/value pairs. The input and output types of the map can be (and often are) different from each other. If the application is doing a word count, the map function would break the line into words and output a key/value pair for each word. Each output pair would contain the word as the key and the number of instances of that word in the line as the value. ===Partition function=== Each ''Map'' function output is allocated to a particular ''reducer'' by the application's ''partition'' function for [[sharding]] purposes. The ''partition'' function is given the key and the number of reducers and returns the index of the desired ''reducer''. A typical default is to [[Hash function|hash]] the key and use the hash value [[Modulo operation|modulo]] the number of ''reducers''. It is important to pick a partition function that gives an approximately uniform distribution of data per shard for [[load balancing (computing)|load-balancing]] purposes, otherwise the MapReduce operation can be held up waiting for slow reducers (reducers assigned more than their share of data) to finish. Between the map and reduce stages, the data is ''shuffled'' (parallel-sorted / exchanged between nodes) in order to move the data from the map node that produced it to the shard in which it will be reduced. The shuffle can sometimes take longer than the computation time depending on network bandwidth, CPU speeds, data produced and time taken by map and reduce computations. ===Comparison function=== The input for each ''Reduce'' is pulled from the machine where the ''Map'' ran and sorted using the application's ''comparison'' function. ===Reduce function=== The framework calls the application's ''Reduce'' function once for each unique key in the sorted order. The ''Reduce'' can iterate through the values that are associated with that key and produce zero or more outputs. In the word count example, the ''Reduce'' function takes the input values, sums them and generates a single output of the word and the final sum. ===Output writer=== The ''Output Writer'' writes the output of the ''Reduce'' to the stable storage. ==Performance considerations== MapReduce programs are not guaranteed to be fast. The main benefit of this programming model is to exploit the optimized shuffle operation of the platform, and only having to write the ''Map'' and ''Reduce'' parts of the program. In practice, the author of a MapReduce program however has to take the shuffle step into consideration; in particular the partition function and the amount of data written by the ''Map'' function can have a large impact on the performance. Additional modules such as the ''Combiner'' function can help to reduce the amount of data written to disk, and transmitted over the network. When designing a MapReduce algorithm, the author needs to choose a good tradeoff<ref name="ullman">{{Cite journal | doi = 10.1145/2331042.2331053| title = Designing good MapReduce algorithms| journal = XRDS: Crossroads, the ACM Magazine for Students| volume = 19| pages = 30| year = 2012| publisher = [[Association for Computing Machinery]]| last1 = Ullman | first1 = J. D. | authorlink1 = Jeffrey Ullman| url = http://xrds.acm.org/article.cfm?aid=2331053 | subscription=yes}}</ref> between the computation and the communication costs. Communication cost often dominates the computation cost,<ref name="ullman"/> and many MapReduce implementations are designed to write all communication to distributed storage for crash recovery. For processes that complete fast, and where the data fits into main memory of a single machine or a small cluster, using a MapReduce framework usually is not effective: since these frameworks are designed to recover from the loss of whole nodes during the computation, they write interim results to distributed storage. This crash recovery is expensive, and only pays off when the computation involves many computers and a long runtime of the computation - a task that completes in seconds can just be restarted in the case of an error; and the likelihood of at least one machine failing grows quickly with the cluster size. On such problems, implementations keeping all data in memory and simply restarting a computation on node failures, or - when the data is small enough - non-distributed solutions will often be faster than a MapReduce system. ==Distribution and reliability== MapReduce achieves reliability by parceling out a number of operations on the set of data to each node in the network. Each node is expected to report back periodically with completed work and status updates. If a node falls silent for longer than that interval, the master node (similar to the master server in the [[Google File System]]) records the node as dead and sends out the node's assigned work to other nodes. Individual operations use [[atomicity (programming)|atomic]] operations for naming file outputs as a check to ensure that there are not parallel conflicting threads running. When files are renamed, it is possible to also copy them to another name in addition to the name of the task (allowing for [[Side-effect (computer science)|side-effects]]). The reduce operations operate much the same way. Because of their inferior properties with regard to parallel operations, the master node attempts to schedule reduce operations on the same node, or in the same rack as the node holding the data being operated on. This property is desirable as it conserves bandwidth across the backbone network of the datacenter. Implementations are not necessarily highly reliable. For example, in older versions of [[Hadoop]] the ''NameNode'' was a [[single point of failure]] for the distributed filesystem. Later versions of Hadoop have high availability with an active/passive failover for the "NameNode." ==Uses== MapReduce is useful in a wide range of applications, including distributed pattern-based searching, distributed sorting, web link-graph reversal, Singular Value Decomposition,<ref>{{cite journal|last1=Bosagh Zadeh|first1=Reza|last2=Carlsson|first2=Gunnar|title=Dimension Independent Matrix Square Using MapReduce|url=http://stanford.edu/~rezab/papers/dimsum.pdf|accessdate=12 July 2014}}</ref> web access log stats, [[inverted index]] construction, [[document clustering]], [[machine learning]],<ref name="mrml">{{cite web| url=http://www.willowgarage.com/map-reduce-machine-learning-multicore| title=Map-Reduce for Machine Learning on Multicore| author=[[Cheng-Tao Chu]]| coauthors=Sang Kyun Kim, Yi-An Lin, YuanYuan Yu, Gary Bradski, Andrew Ng, and [[Kunle Olukotun]]| publisher=NIPS 2006}}</ref> and [[statistical machine translation]]. Moreover, the MapReduce model has been adapted to several computing environments like multi-core and many-core systems,<ref name="evalMR">{{Cite book | doi = 10.1109/HPCA.2007.346181| chapter = Evaluating MapReduce for Multi-core and Multiprocessor Systems| title = 2007 IEEE 13th International Symposium on High Performance Computer Architecture| pages = 13| year = 2007| last1 = Ranger | first1 = C. | last2 = Raghuraman | first2 = R. | last3 = Penmetsa | first3 = A. | last4 = Bradski | first4 = G. | last5 = Kozyrakis | first5 = C. | isbn = 1-4244-0804-0}}</ref><ref name="graphicsMR">{{Cite book | doi = 10.1145/1454115.1454152| chapter = Mars: a MapReduce framework on graphics processors| title = Proceedings of the 17th international conference on Parallel architectures and compilation techniques - PACT '08| pages = 260| year = 2008| last1 = He | first1 = B. | last2 = Fang | first2 = W. | last3 = Luo | first3 = Q. | last4 = Govindaraju | first4 = N. K. | url = http://wenbin.org/doc/papers/Wenbin08PACT.pdf| last5 = Wang | first5 = T. | isbn = 9781605582825}}</ref><ref name="tiledMR">{{Cite book | doi = 10.1145/1854273.1854337| chapter = Tiled-MapReduce: optimizing resource usages of data-parallel applications on multicore with tiling| title = Proceedings of the 19th international conference on Parallel architectures and compilation techniques - PACT '10| pages = 523| year = 2010| last1 = Chen | first1 = R. | last2 = Chen | first2 = H. | last3 = Zang | first3 = B. | isbn = 9781450301787}}</ref> desktop grids,<ref name="gridMR">{{Cite book | doi = 10.1109/3PGCIC.2010.33| chapter = Towards MapReduce for Desktop Grid Computing| title = 2010 International Conference on P2P, Parallel, Grid, Cloud and Internet Computing| pages = 193| year = 2010| last1 = Tang | first1 = B. | last2 = Moca | first2 = M. | last3 = Chevalier | first3 = S. | last4 = He | first4 = H. | last5 = Fedak | first5 = G. | url = http://graal.ens-lyon.fr/~gfedak/papers/xtremmapreduce.3pgcic10.pdf| isbn = 978-1-4244-8538-3}}</ref> volunteer computing environments,<ref name="volunteerMR">{{Cite book | doi = 10.1145/1851476.1851489| chapter = MOON: MapReduce On Opportunistic eNvironments| title = Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing - HPDC '10| pages = 95| year = 2010| last1 = Lin | first1 = H. | last2 = Ma | first2 = X. | last3 = Archuleta | first3 = J. | last4 = Feng | first4 = W. C. | last5 = Gardner | first5 = M. | last6 = Zhang | first6 = Z. | url = http://eprints.cs.vt.edu/archive/00001089/01/moon.pdf| isbn = 9781605589428}}</ref> dynamic cloud environments,<ref name="dynCloudMR">{{Cite journal | doi = 10.1016/j.jcss.2011.12.021| title = P2P-MapReduce: Parallel data processing in dynamic Cloud environments| journal = [[Journal of Computer and System Sciences]]| volume = 78| issue = 5| pages = 1382| year = 2012| url = http://grid.deis.unical.it/papers/pdf/MarozzoTaliaTrunfioJCSS2012.pdf| last1 = Marozzo | first1 = F. | last2 = Talia | first2 = D. | last3 = Trunfio | first3 = P. }}</ref> and mobile environments.<ref name="mobileMR">{{Cite book | doi = 10.1145/1839294.1839332| chapter = Misco: a MapReduce framework for mobile systems| title = Proceedings of the 3rd International Conference on PErvasive Technologies Related to Assistive Environments - PETRA '10| pages = 1| year = 2010| last1 = Dou | first1 = A. | last2 = Kalogeraki | first2 = V. | last3 = Gunopulos | first3 = D. | last4 = Mielikainen | first4 = T. | last5 = Tuulos | first5 = V. H. | isbn = 9781450300711}}</ref> At Google, MapReduce was used to completely regenerate Google's index of the [[World Wide Web]]. It replaced the old ''ad hoc'' programs that updated the index and ran the various analyses.<ref name="usage">{{cite web| quote=As of October, Google was running about 3,000 computing jobs per day through MapReduce, representing thousands of machine-days, according to a presentation by Dean. Among other things, these batch routines analyze the latest Web pages and update Google's indexes.| url=http://www.baselinemag.com/c/a/Infrastructure/How-Google-Works-1/5| title=How Google Works| publisher=baselinemag.com}}</ref> Development at Google has since moved on to technologies such as Percolator, Flume and MillWheel that offer streaming operation and updates instead of batch processing, to allow integrating "live" search results without rebuilding the complete index.{{citation needed|date=February 2015}} MapReduce's stable inputs and outputs are usually stored in a [[distributed file system]]. The transient data is usually stored on local disk and fetched remotely by the reducers. ==Criticism== ===Lack of novelty=== [[David DeWitt]] and [[Michael Stonebraker]], computer scientists specializing in [[parallel database]]s and [[shared-nothing architecture]]s, have been critical of the breadth of problems that MapReduce can be used for.<ref name="shark">{{cite web| url=http://typicalprogrammer.com/?p=16| title=Database Experts Jump the MapReduce Shark}}</ref> They called its interface too low-level and questioned whether it really represents the [[paradigm shift]] its proponents have claimed it is.<ref name="ddandms1">{{cite web| url=http://craig-henderson.blogspot.com/2009/11/dewitt-and-stonebrakers-mapreduce-major.html| title=MapReduce: A major step backwards| author=[[David DeWitt]]|author2=[[Michael Stonebraker]] | publisher=craig-henderson.blogspot.com| accessdate=2008-08-27}}</ref> They challenged the MapReduce proponents' claims of novelty, citing [[Teradata]] as an example of [[prior art]] that has existed for over two decades. They also compared MapReduce programmers to [[CODASYL|Codasyl]] programmers, noting both are "writing in a [[Low-level programming language|low-level language]] performing low-level record manipulation."<ref name="ddandms1"/> MapReduce's use of input files and lack of [[Logical schema|schema]] support prevents the performance improvements enabled by common database system features such as [[B-tree]]s and [[Partition (database)|hash partitioning]], though projects such as [[Pig (programming language)|Pig (or PigLatin)]], [[Sawzall (programming language)|Sawzall]], [[Apache Hive]],<ref name="ApacheHiveWiki">{{cite web| url=https://cwiki.apache.org/confluence/display/Hive/Home| title=Apache Hive - Index of - Apache Software Foundation}}</ref> [http://ysmart.cse.ohio-state.edu/ YSmart],<ref name="YSmartPaper">{{cite web| url=http://www.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-11-7.pdf| title=YSmart: Yet Another SQL-to-MapReduce Translator| author=Rubao Lee, Tian Luo, Yin Huai, Fusheng Wang, Yongqiang He and Xiaodong Zhang|format=PDF}}</ref> [[HBase]]<ref name="HBase">{{cite web| url=http://hbase.apache.org/| title=HBase - HBase Home - Apache Software Foundation}}</ref> and [[BigTable]]<ref name="HBase"/><ref name="BigTablePaper">{{cite web| url=http://research.google.com/archive/bigtable-osdi06.pdf| title=Bigtable: A Distributed Storage System for Structured Data| format=PDF}}</ref> are addressing some of these problems. Greg Jorgensen wrote an article rejecting these views.<ref name="gj1">{{cite web| url=http://typicalprogrammer.com/?p=16| title=Relational Database Experts Jump The MapReduce Shark| author=[[Greg Jorgensen]] | publisher=typicalprogrammer.com| accessdate=2009-11-11}}</ref> Jorgensen asserts that DeWitt and Stonebraker's entire analysis is groundless as MapReduce was never designed nor intended to be used as a database. DeWitt and Stonebraker have subsequently published a detailed benchmark study in 2009 comparing performance of [[Hadoop|Hadoop's]] MapReduce and [[RDBMS]] approaches on several specific problems.<ref name="sigmod">{{cite web| url=http://database.cs.brown.edu/projects/mapreduce-vs-dbms/| title=A Comparison of Approaches to Large-Scale Data Analysis| author=Andrew Pavlo| coauthors=E. Paulson, A. Rasin, D. J. Abadi, [[David DeWitt|D. J. Dewitt]], S. Madden, and [[Michael Stonebraker|M. Stonebraker]]| publisher=Brown University| accessdate=2010-01-11}}</ref> They concluded that relational databases offer real advantages for many kinds of data use, especially on complex processing or where the data is used across an enterprise, but that MapReduce may be easier for users to adopt for simple or one-time processing tasks. Google has been granted a patent on MapReduce.<ref name="patent">[http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL&p=1&u=/netahtml/PTO/srchnum.htm&r=1&f=G&l=50&s1=7,650,331.PN.&OS=PN/7,650,331&RS=PN/7,650,331 US Patent 7,650,331: "System and method for efficient large-scale data processing "]</ref> However, there have been claims that this patent should not have been granted because MapReduce is too similar to existing products. For example, map and reduce functionality can be very easily implemented in [[Oracle database|Oracle's]] [[PL/SQL]] database oriented language<ref name="Curt Monash">{{cite web| url=http://www.dbms2.com/2010/02/11/google-mapreduce-patent/| title=More patent nonsense — Google MapReduce| author=[[Curt Monash]] | publisher=dbms2.com| accessdate=2010-03-07}}</ref> or is supported for developers transparently in distributed database architectures such as [[Clusterpoint]] XML database<ref name="Clusterpoint">{{cite web| url=http://www.clusterpoint.com| title=Clusterpoint XML database| publisher=clusterpoint.com}}</ref> or [[MongoDB]] NoSQL database.<ref name="MongoDB">{{cite web| url=http://www.mongodb.org | title=MongoDB NoSQL database| publisher=10gen.com}}</ref> ===Restricted programming framework=== MapReduce tasks must be written as acyclic dataflow programs, i.e. a stateless mapper followed by a stateless reducer, that are executed by a batch job scheduler. This paradigm makes repeated querying of datasets difficult and imposes limitations that are felt in fields such as [[machine learning]], where iterative algorithms that revisit a single [[working set]] multiple times are the norm.<ref>{{cite conference |first1=Matei| last1=Zaharia| first2=Mosharaf |last2=Chowdhury| first3=Michael| last3=Franklin| first4=Scott| last4=Shenker| first5=Ion| last5=Stoica |title=Spark: Cluster Computing with Working Sets |titlelink=Spark (cluster computing framework) |conference=HotCloud 2010|date=June 2010}}</ref> ==Conferences and users groups== * [http://graal.ens-lyon.fr/mapreduce/ The First International Workshop on MapReduce and its Applications (MAPREDUCE'10) ] was held in June 2010 with the HPDC conference and OGF'29 meeting in Chicago, IL. * [http://mapreduce.meetup.com/ MapReduce Users Groups ] around the world. ==See also== ===Implementations of MapReduce=== * [[Apache Hadoop]] * [[Couchdb]] * [http://discoproject.org Disco Project] * [[Infinispan]] * [[Riak]] ==References== {{Reflist|3}} ==External links== {{Commons category|MapReduce}} {{Google Inc.}} {{DEFAULTSORT:Mapreduce}} [[Category:Google software]] [[Category:Parallel computing]] [[Category:Distributed computing architecture]] [[Category:Articles with example code]]'