Data Science for beginners full course part 1

Foreign [Music] Welcome to Introduction to Data Science. My name is Bill Howe, and I'm the director of research for scalable data analytics at the University of Washington e-Science Institute and an affiliate assistant professor in computer science and engineering, also at the University of Washington.

So in this first segment, what I want to do is go through some examples of data science activities and projects from the recent past that I found interesting and use them to sort of whet your appetite for the concepts that we're going to learn in this course. Okay, so the first one I want to mention here is the presidential election from 2012. And I know you're probably sick of hearing about this if you're if you live in the United States, and even if you don't, you may be sick of hearing about it, but bear with me.

So this is a map of the Electoral College, and each state is colored for the candidate that took the electoral votes, and the numbers represent how many electoral votes each state has. And so if you recall, what was interesting about this map at the time was that it was uh or that it led to a pretty significant discussion in the media about data science because Nate Silver of the 538 Blog was able to predict this map perfectly before the election. All right, and you know that discussion in the media had talked a lot about um, you know, what a genius Nate Silver was and mentioned the sophisticated mathematics he was using and how, you know, it's he's sort of a whiz with these things. But what I thought was interesting about this was that Nate Silver would be the first one to tell you that the methods he was employing to make this prediction were actually pretty simple. Right.

And so he says here in a series of quotes from blog posts around that time, this first one from October 26th was, "The innovation behind this ought to be very simple: Mr. Obama is maintaining leads in the polls in Ohio and other states that are sufficient for him to win 270 electoral votes." And then that was funny was, you know, a few days later, uh, he becomes sort of more blunt: "The argument we're making is exceedingly simple: here it is, Obama's ahead in the hot in Ohio. Right, it's not a magic trick." And so then after the election, on November 10th, when he was shown to be right and got this sort of flawless prediction, uh, the blog post that this last quote is taken from—whoops, excuse me—the last quote was taken from was describing why he started the 538 Blog in the first place, and he says, "Look, you know, the bar set by the competition was invitingly low. Someone could look like a genius simply by doing some fairly basic research into what really has predictive power in a political campaign." And so what really had predictive power in this case was the state polls themselves, aggregated. Right. So historically, the state polls aggregated uh did a pretty good job of predicting the outcome of the general election, and so that's what he did. Now there was some sophisticated work in quantifying the uncertainty and certainly in presenting these results; there's a lot of beautiful interactive visualizations that he created in order to sort of convey these ideas to the public. And that's one of the points I want to make about this is that getting the answer in some cases is the easy part; it's in interpreting the results and in convincing others of the result by presenting them, usually through visualizations, that can be the hard part. And this is one of the themes that we'll come back to throughout this course. Okay.

So just to summarize that, though I'm not sure I said this: simple methods plus enough good data, you know, wins; that trumps more sophisticated methods in many cases, and that's another theme that we'll come back to. All right. So something else related to the campaign before we moved on to other topics was the system that the Obama campaign used for their data-driven ground game, so to speak: the ability to sort of target, direct, uh, you know, target specific categories of users. And so what they did was they built and maintained a really significantly sized, a massive voter database and used it to design these highly tailored messages to very, very specific groups. So, you know, the mother of two in a small town in Ohio who tweeted about the environment and mentioned organic vegetables on her Facebook page and, you know, who had voted in 2008 and had registered on Obama's website but had never donated, okay, you know, she would get a message from Michelle Obama that highlighted uh Barack Obama's environmental policies. Okay. And so in order to design these messages, what you had to do was kind of do kind of ad hoc hypothesis testing about what might work and what didn't; you had to kind of slice and dice this data at kind of interactive speeds. And this is another theme that we'll return to is the need for these kind of ad hoc interactive analyses, and the systems they use for this are pretty interesting, too. You know, this was a SQL database, a very fast one called Vertica, and we'll talk a little bit about what makes Vertica special I hope toward the end of the course, but it is a SQL database. And so SQL sometimes gets a bad name in data science context as sort of the Old Guard that can't be possibly used for analytics and doesn't really make sense in today's era, but, you know, don't don't believe it. Right, it has a role to play in many in many cases. And so here, you know, they did use Hadoop right to do the aggregate generations of anything not real time, he says here, but for the speed of thought queries about the data, they used this Vertica database. Okay, and so we'll come back to systems uh in a several segments from now.

Okay, so moving on—this was around the same time—you know, when Hurricane Sandy made landfall, one of the things that struck me was the fact that visualizations of available data were starting to emerge in real time in response to the storm. And there's some very nice examples of people who used Twitter data to analyze or to produce a map of where the power was going out. And in this even simpler case, Joseph Ruhl got public data from local weather stations and just from two different weather stations and just simply plotted them. And so this is the barometric pressure over the course of this, you know, few-day period—two days, I guess—in two locations: Atlantic City and Philadelphia. And so you can see this enormous dip as the storm passing through, and you can also see the time lag between Atlantic City and Philadelphia, and you can also see the intensity is probably a little bit higher given that the barometer drop is more significant in in Atlantic City. Okay. So a couple things here: one is pulling data down from the web and repurposing it sort of in real time or at least in in short time—maybe not real time—to produce visualizations and then publishing those back on the web. I think this is very much the character of data science activities in this particular example. There's not necessarily a large data set involved, but repurposing data that was collected for a different purpose is is a thing that we'll come back to, and again we see the ad hoc nature of this as well. Fine. So another plot here's wind speeds, and they sort of peak out at 40 in Atlantic City, which is still the green, and you can see Atlantic City is indeed more intense here, and the gray here is error bars. So again, another variant on the same same data.

Okay, so changing gears a little bit, this was a study—the title here is called "The Expression of Emotions in 20th-Century Books"—and so what they were interested in is: have the words that we choose to use in our collective literature changed over time, and eventually does that tell us something about sort of culture or civilization? I I find the scientific inquiry sort of compelling, but what I think is most striking about this and why I want to include this example is that the methodology that they used here is pretty straightforward; you could you could do this yourself with not a significant background in either technology or in statistics or even in uh, you know, linguistics or anything. And so this is what they did. Right. So the first step is kind of a doozy: this is take all the books written in the 20th century and digitize them. Well, that would be a non-starter; none of us could do that, but that's okay; Google's already done it for us, and they've made the data available at this URL, and so you can go check that out. What they've done is digitize the books, done the character recognition on it, and produced these n-gram data sets. So these are tables of of data where each row has an n-gram and followed by the year and the counts of the number of times that n-gram occurred. This has already been broken down and processed into a form that's digestible. Okay. And so what's an n-gram? Well, it's pretty simple: a one-gram is just a single word, like "yesterday"; a five-gram, an example here is the phrase "analysis is often described as." And so in this study, they just ignored everything but the one-grams, uh, and then they took some sense as a some subset of those one-grams and assigned them a mood score. So how do they do this? Well, you can imagine that certain words are charged with a particular mood or associated with joy or sadness or fear and so on, and you could also imagine that synonyms of those words might also be associated with that mood. And so this analysis sounds non-trivial, and it is, but once again, that's already been done for you; there's a a resource on the web called WordNet where they've done this kind of affect analysis. And so the authors of this paper were able to take the digitized books from Google—or already broken down into n-grams—and the affect scores from WordNet and then do this calculation, which, you know, may sort of look intimidating if you're not used to staring at these mathematical expressions, but it's actually pretty simple: this is the count of a particular uh word in the set—the set being the set of WordNet words, which is not as big as all the words; only only some words are able to be scored as mood—and then you normalize by the count of the number of occurrences of the word. The so why do they do that? Well, you need to normalize over something in order to account for the fact that perhaps we just write more books in 2005 than we did in 1937, or we've been able to digitize more books, so we need to normalize by that total. Well, why not just normalize by the total number of words? Well, the reason is that the the word does a better indicator of prose than the total number of of words, and this is because we've also apparently we've also started to produce more sort of captions and figures and more sort of technical language and more sort of formula and more expressions, more non-prose uh utterances in these in these uh books, and therefore we can sort of skew the results. So they really want to capture that in our language: when we actually write full complete sentences, how often are these words being used? Okay. And then you add those up, and you divide by the uh total number of words in the set, and then there's one more transformation here that should look familiar to you if you sort of recall your high school statistics. And so um, you subtract the mean and divide by this standard deviation. So this is normalizing with respect to a normal distribution. You know, there's a count, and there's a division, and then there's two data sets that you can pull from the web, and they're big, but they're not exceedingly varied; they fit in memory on most of your laptops nowadays. So it's a significant computational task, but nothing that requires a—you could do this in a weekend if you had thought about it. Okay. So I find that pretty compelling. Fine.

So these are the results: this is joy words minus sadness words; this is the z-score for joy and sadness, and you can see the sort of a big dip after World War uh Two, and that's one of the points they make in the in the paper, and then you can see this sort of thing start to increase in the late 90s. So I won't try to analyze this for the scientific value; I'll just present the results, but I think is maybe more interesting is this one. So this is now emotion words total minus random words total, and there's a sort of prominent downward slope over time. So what is this? What is this? What's going on here? Well, apparently you can make the argument that we're using fewer emotion words over time. Okay, that said, there's a bit of an uptick in this red line, and so what does that represent? Well, that's fear words, and you know you can imagine some of the reasons why there might be an increase in fear words since the since the 1980s. Okay. So this is pretty fun, though. This is a significant analysis; it can be done just by taking these these data sets that they didn't have to prepare themselves. And then the other point I'll make about this—this is just a copy and paste of a segment of some of the papers that this paper cites—and I just was struck by the titles here: you know, "Quantitative analysis of culture using millions of digitized books," "Quantifying the evolutionary dynamics of language," "Frequency of word use predicts rates of lexical evolution through Indo-European history," "Song lyrics and linguistic markers." I mean, what strikes me about this is that, you know, linguistics, anthropology, history, you know, culture—these studies are becoming hard sciences by the virtue of data-driven methods. Right. So all science is becoming data science. Right. And therefore data scientists have a lot of power in this regime; it's a great time to be, you know, a data geek. Okay. You know, there's data journalism as well. I mean, one point I probably should put a slide in here about this, but you know, when the WikiLeaks material came out, you know, you're not going to pour yourself a pot of coffee and pour over those materials, you know, print them all out and sort of go through them one by one; you're going to write algorithms that do this kind of an analysis, you know, word-use analysis, look for email chains and dialogues—these sort of computational methods in order to analyze that material. And so now journalism itself is a computational enterprise; it's a data science problem, not a uh or at least it's amenable to data science techniques. So, you know, as a data scientist, the world is your oyster. All right. So let me pause there, and we'll pick up with a couple more examples before moving on in the next segment. [Music]

So let's go through a few more examples. So if you were asked to decide how important a particular scientific paper was relative to other papers, how much you go about doing that? Well, one way to decide between, say, these two papers here [Music]—I'll mark one with a blue circle and one with a red circle—is to wait until other papers start to cite these and count up the number of citations. So here, you know, the paper marked with the blue circle has had four other papers—other authors have bothered bothered to read this paper and cite it—so them therefore must have had more impact in the scientific community than the one marked with the red circle. But if we wait even longer—and that's indicated here by this darker blue color—if we wait even longer, you might get even more people citing this intermediate paper, and so maybe we can conclude that well, over time, ultimately this one had more impact because it, you know, this this paper was influenced by this paper, and all of these papers were influenced by this one. So therefore, perhaps we change our answer and that this one is more important. So how do we decide between these two interpretations? Well, this problem looks a lot like the problem of judging the relative importance of pages on the web. And so one thing you can do is say, "Well, look, you know, a particular website is important if a lot of other important websites point to it." And so here you can say a scientific paper is important if a lot of other important papers point to it. And this method that Google proposed and implemented and ultimately led to a pretty significant part of their success was this algorithm, PageRank. And so Pedro did exactly this: you add up all the weights of your neighbors and give them to yourself, and then pass on that weight to everybody that you link to, and you keep going with this until you reach some convergence condition, and you end up with the relative importance of these uh of these pages. Okay. And so this is a pre this is a method that comes up quite a bit whenever you have a graph; it makes sense to potentially run a PageRank algorithm on it, even though it had nothing to do with uh ranking things on the web, for which it was originally designed.

Okay. Using the same data set, you can—Carl Bergstrom and Martin Rosvall created this visualization. So ignore the importance question; just think about the graph of the citation network and doing analytics on it. One kind of analysis you can do is judging importance; another kind of analysis you can do is this. And so what this is is: over time, run a clustering algorithm—which I haven't described what that is, but it groups clusters of similar documents—to and then map those clusters into the rel these fields. And so you can determine that with some problem with with some confidence that this cluster represents medicine because it has the Lancets and it has other kinds of medical journals in it, and you can conclude that this cluster is represents molecular and cell biology. But what's pretty striking is that if you lay this out in this timeline like this, you can see that some fraction of the molecular and cell biology community and the neurology community started to combine to form a brand new field of science called neuroscience. And so just by doing this kind of analytics on this graph, right, to do this data science on this graph, you can uncover the emergence of new fields of science. And so I found that pretty striking, and they've gone on to do many other kinds of analysis on this same data set, and the overall field of this kind of meta-field of studying the scientific literature in order to draw inferences is called bibliometrics. Pin is a little wonky bibliometrics; sorry for the bad B there. Okay.

So what can't what is not amenable to data science? Well, you know, you might think food. And this is a paper in a fairly respectable journal that uh applied some data-driven techniques to analyzing food pairing. Okay. And so what they did here is I'm pretty pretty interesting. Right. So induce a graph on the ingredients by saying that if two ingredients appear together in a recipe, you draw an edge between them—a graph meaning vertices and edges—and if you've never really heard—if you're not familiar with graphs—you will be by the end of the course, but bear with me right now. So connect two ingredients if they appear together in some recipe. Okay, build that big graph, and now you can analyze it in ways that are similar to what we just talked about; you can look at the community structure; you can find the clusters within this graph and see if these clusters correspond to some of the well-known methods of uh food pairings, and in some cases they do; in some cases they don't, and the author sort of show that they've uncovered things that were not necessarily known but appear to be there in the data. Right. And so this data-driven approach, you know, the fact that there's websites full of recipes online has allowed us to put things on on a more quantitative basis that were previously simply, you know, old wives' tales, apparently, essentially. Okay. So I thought that was kind of a fun example of an unusual use of data science, interesting again in a respectable journal. All right. So another example: this is from The Last.fm blog; they looked at the tags associated with the songs on Last.fm and usually do some simple analysis of the emergence of genres over time, but you know, based on the popularity of the tags for songs coming from that time. And so you see things like, well, Post-Punk in red here came after Punk in purple, which you'd hope the graph shows that you'd expect; you'd also see some kind of a a rise here in rock and roll over time and then maybe a bit of a a dip more recently. And so that's kind of interesting, but the the other theme here we have is uh that we mentioned in the in the previous segment is repurposing data. Right. So this data was collected simply to help

With search, you know, find similar music, and it's now being reused to sort of draw inferences about the emergence of, uh, entire genres. Okay. Okay. So another example here that you may be familiar with: Google was able to show that by analyzing the search logs, the frequency of search terms, it could do a better job predicting the severity and the scope of flu outbreaks than the Centers for Disease Control. And by "better" here, we mean essentially earlier, right? It was able to, uh, give more of a head start to the health community. Okay. And also, it was sort of more accurate. So how do they do this? Well, you know, when you're getting the flu, it turns out that you want to search for flu symptoms, terms associated with flu symptoms, more often. And by watching that uptick, you can predict that there's that the flu outbreak is coming. Okay. So that's great, and they put in a, they this worked, and they sort of published a paper out and about it, and then they put up this interactive visualization that a lot of people to sort of analyze celebrity going forward. But just this year, you know, some folks showed that it didn't do a very good job in this last year. So scientific hindsight shows that the Google Flu Trends far overstated this year's flu season, and the reason they think explains this is that there was lots of media attention associated with this year's flu season because it was it was a bit of an uptick, and so it got amplified. And so this caused people to search for flu, flu-related terms more often, perhaps because they're worried about experiencing the symptoms, but perhaps because they are trying to understand more about the flu outbreak, perhaps because they're searching for articles, perhaps because they're worried about their kids more, but it all was it was a second-order effect based on the media attention on the problem, which led to skewed results and ultimately a wrong answer. And so the point here is this is great that they're repurposing data from the search engine to try to make predictions about something else, but it's it's by it's it is biased data, and so you have to be careful with what you conclude from it. Okay. So there are limits here. All right. So another example, also also analyzing [Music] um, web search traffic that was done with perhaps a little bit more of a scientific rigor was done by some folks at Microsoft Research. And so here what you're looking for is side effects associated with particular drugs. And so these results are pretty straightening. So what this graph is showing that is over time a set of users, it was around about a million, that had they had permission to sort of monitor their their web search traffic, that when you searched for this drug in green, what percentage of the time did you also search for terms associated with hyperglycemia symptoms? And the answer is somewhere around five percent. For this other drug, it was somewhere around four percent. For the in the background for the average case, it was pretty close to zero percent. If you search for both of these drugs, the odds that you also search for hyperglycemia symptoms went up to 10 percent. Okay. So what's striking about this is that hyperglycemia is not a known side effect of these drugs, but it seems impossible to ignore from the web search data. Right? There's just no reason to believe that this could be explained by a coincidence, and they developed this argument more in the in the paper than I than I just have there. But so fine. So repurposing data, this is another example of a large data system they had to use a web, uh, uh, search, um, and those are probably the two points I want to make about that, but a pretty fun one. Okay. So the last example I'll give is a different take; there's more more about prediction than data, but this from last October. If you recall, there were six Italian seismologists who were convicted of manslaughter for failing to predict a magnitude 6.3 earthquake in April 2009. And so while the locals were concerned about the size and activity, the researchers were deemed to be just too reassuring about the verdict. And so the point I want to make here is that you know, there's liability, uh, I mean, so this this this the scientific community was completely aghast that has happened, and I'm completely against this happened, and pretty much everybody is that you can imagine to hold researchers responsible for failing to predict something that is demonstrably and known to be impossible to predict. Right? So there's no seismologist on the planet that would argue that earthquakes are even remotely predictable, and yet the courts sort of decided that somehow they, you know, because they got the wrong answer, it's bad. But it does sort of bring up the issue that when you make a prediction, there's a certain amount of weight you're going to put behind it, whether whether intentionally or not. And so understanding how confident you are about that prediction is sort of an important part of the game here. Okay. So that was the last example. The themes that we saw come out here: we gave a couple of examples of graph analytics; we showed that databases are sort of useful in the Obama grounding case; so a lot of examples of visualization and communicating the results; interpreting these results; we saw some examples of using very large data sets; other examples that use very small data sets; and not everything's about Big Data. A couple of bullets that aren't on here: we talked about, you know, ad hoc interactive analysis is sort of not just faster but different, and so supporting that is important. And then we talked about repurposing data, right? So data collected perhaps by someone else for some other purpose, reusing that to draw inferences about something else; that's a pretty common theme. Here in the next couple of segments, we'll talk about how we organized this course and some of the design decisions we made in creating the material.

Welcome back. So I want to talk about in this segment what this term "data science" actually means. So you know, you'll see these quotes around the web. So in Fortune Magazine, you talk about data science being the hot new gig in tech, and how Varian, who is Google's Chief Economist, in the New York Times in 2009, which is a while ago now, talked about statistics being the next sexy job and described it as, you know, the ability to take data, to understand it, process it, extract value from it, and communicate it; that's going to be hugely important. Another person who's prolific in thinking and writing in this space is Mike Driscoll, who's the CEO of a company called Metamarkets, and he talks about they're the data science, you know, as it's practiced is this, you know, there's this colloquial view of it is a blend of Red Bull-fueled hacking and espresso-inspired statistics. And maybe another quote of his is the data sciences that the civil engineering of data, who, you know, whose acolytes possess practical knowledge as well as a theoretical understanding. And so there's a balance between pragmatism and theory is something we'll come back to. So another perspective on this that you should be familiar with is this Venn diagram that made the rounds several years ago by Drew Conway, and what he's his point was that data science is perhaps the mix of three different sort of areas: one is hacking skills, you know, programming expertise; another is the academic view, you know, the math and statistics knowledge; and then the third that he added is this notion of substantive expertise. And what you mean by this is kind of a deep investment with the data. So if you think about your typical IT shop, they're typically building tools for other people to use to actually analyze the data, but they don't necessarily do the analysis themselves. And in, you know, a data scientist, in contrast, may be a participant in tool building, but they're going to also sort of dive deeply into the data and do the analysis themselves. And that's one of the ways that I like to interpret Drew's blue bullet here of substituted expertise. Okay. And then he also sort of fills in the gaps between these, talking about that, uh, I'm not sure I'd agree that this is necessarily traditional research, but perhaps it is, is applying the statistical knowledge to a particular domain is where research comes in, and that some deep theory plus some pragmatic programming issue a machine learning expert. And then he sort of refers to this as the danger zone where you sort of know enough about the domain to be dangerous and you know enough hacking to be dangerous, but you don't know how to ground your analysis in proper theory. And you know, some of my colleagues like to joke that computer scientists don't understand error bars, right? And that's that's maybe what they're referring to here. So fine. So what do data scientists actually do? Well, there's some more quotes here from from EMC, who a company who acquired a company called Greenplum and have recently have a pretty significant initiative in data science, both related to a product called Greenplum as well as other products. So they need to find the nuggets of truth in data and then explain it to business leaders. And I like this, you know, and then explain it to business leaders; somebody else will come back to that. I mentioned previously is that communicating the results is a critical part of this process; it's not just getting the result. Okay. And then another view of this, which is sort of interesting, is that data scientist DJ Patel talks about data scientists tend to be hard scientists, maybe coming from a physics background, who have a strong mathematical background and computing skills and, you know, come from a discipline in which survival depends on getting the most from the data; they're really used to sort of torturing the data to extract every last ounce of value out of it. This, you know, DJ has an applied math background, so he's maybe coming from that perspective. So Mike Driscoll also talks about the three sexy skills of data geeks, which are statistics, this thing, data munging, and we'll kind of, you know, this funny word "munging," where we kind of return to this; you'll see a lot of this sort of colloquial language, and I'll give my perspective on what I think that tells us about, uh, the the state of the world in data science. And but what he means by this, as you can imagine, is sort of parsing data and scraping data from the web and converting to between different file formats efficiently and not getting hung up on these, uh, the kind of friction that you deal with when you are working with large and and heterogeneous data sets. Right? So a data scientist is someone who's very comfortable in that environment, is able to sort of work nimbly even when things aren't very clean. Okay. And then finally, visualization is another sexy skill, right? The ability to communicate the results through visualization. So another quote from Jeffrey Stanton, who teaches a course in data science at Syracuse and has and is involved in one of the earlier programs in data science, talks about the emerging area of work concerned with the collection, preparation, analysis, visualization, management, and preservation of large collections of information. And I think one thing that's interesting about his perspective is includes his word "preservation." So while the, you know, preparation, the analytics, and the visualization are the three tenants that you see quite often, that's something that we like in this course as well. You know, Jeffrey goes one step further and talks about preservation. So even after you're done communicating the results, what do you do with the data long term? And that's part of the reason he's got a background in library science and and information studies where they're very concerned with the curation of data. And so that's something we're probably not going to emphasize too much in this course, but it's definitely part of the overall data life cycle, if you will. Okay. So another quote from a thinker in this space is Hillary Mason, the chief scientist at Bitly. And so she says a data scientist is someone who can obtain, scrub, explore, model, and interpret data, you know, blending hacking, statistics, and machine learning. So we saw that before in Drew Conway's diagram, this blending, and data scientists are not only adept at working with data but can appreciate data itself as a first-class product. And I think she's talking about there maybe is being able to organize the data and actually produce something that's usable by other people, right? Okay. So having a quality data resource, a data asset that others can use to answer questions is one of the outputs of data science. And then, you know, she talks about, uh, let's see, scrubbing, and we've probably saw "munging" on the previous slide, and so you'll see some of these terms; you'll see "data Jiu Jitsu"; people will say, you know, this this the data scientists that I want to hire is really good at data Jiu Jitsu, and we don't I I I'll in a couple of segments I'll talk about or the next segment I think I'll talk about what I think this means. Okay. So to summarize what we've talked about, there's perhaps three overarching tasks involved in data science, and those are, you know, preparing to run some sort of statistical analysis, actually running that statistical analysis, and then interpreting and communicating the results. And this phase here, that's preparing to run the model, is where we see all that data munging, cleaning, manipulating, integrating, and so on. All right. So another view that I like to take is that data science is really about data products, producing data products that you may use yourself or that other others may use. Okay. And so what do I mean by data products? Well, this could be data-driven applications. So if you think about a spell checker, right, this is not just a piece of code, this is not just a piece of software that does something; this is only enabled by a dictionary of words and a dictionary of misspelled words actually. Okay. And similarly, a machine translation where you can type a sentence in French and have it automatically translate it into Arabic relies on not just clever algorithms but rather an enormous corpus of French text and Arabic text. Okay. A second kind of data product is interactive visualizations. We saw this example with the Google Flu application, and there's a suite of visualizations on the web that I'm not going to show right now called that are associated with the global burden of disease, and these are visualizations produced by the Institute for Health Metrics here at the University of Washington. And what I like about it is it's an example of where they've done a lot of research, but then the output was not just a research paper, although they wrote plenty of those as well; it was these interactive visualizations allowed you to explore the data as well. And so this is I think captures this notion of producing not just the answer or not just a paper in perhaps this case but a data product that is usable by others. Okay. And then finally, another kind of data product might be an online data database of some kind that others can actually use to can query and answer their own questions. And so maybe there's not necessarily a visualization component, but the work that goes into producing these things I would argue is part of data science. And so this captures some of the enterprise data warehouse work and software and effort of which there's a lot, including business intelligence work. And a couple of segments I'll try to differentiate those two. Another example of an online database from science as opposed to business is the Sloan Digital Sky Survey, which I'll talk about in more detail in a couple of segments. So again, just to summarize here what's in red: data science is not just about building data products; it rather is about the building data products, not just answering the questions once. And what data products are are assets, digital assets that empower others to use the data in new ways. And so they may help communicate results, for example, Nate Silver built these maps, or they may empower others to do their own kind of analysis, say with a data warehouse or with a visualization. All right. [Music] So let's talk a little bit about what distinguishes the term "data science" from other related fields. So one related field is business intelligence. Business intelligence systems are associated with a couple of concepts: one is a data warehouse, and the other is a set of dashboards or reports that consume data from the data warehouse and are used to answer particular questions. So both of these components require a lot of upfront effort to design and build and are therefore not too adaptable when requirements change. Okay. And so therefore, a software stack designed for business intelligence may or may not be appropriate for any particular data science problems where changing requirements are considered the norm, and so sort of warrants a new term is that business intelligence became associated with a particular approach to a particular set of problems, and a data science is in some sense broader. Okay. The other point I like to make about business intelligence is that the BI engineers are not typically expected to consume their own data products and perform their own analysis and make and make the business decisions themselves; usually they're building tools for others to make decisions with. Okay. As a data scientist, you'll be doing both. So what about statistics? Well, statistical methods are at the heart of what a data scientist does day to day, but a statistician will typically be comfortable with the with assuming that any data set they encounter will fit in main memory on a single machine. And this makes sense because the whole field was born out of the need to extract the most information possible from a very sparse, very expensive to collect, and and typically therefore very small data set. Okay. So if you only have 20 patients in the world with a particular disease, you can't just go find 20 more cheaply. So therefore, you need to come up with new mathematics to squeeze as much information as you can out of the 20 you already have. But that's not always the problem anymore, right? So as we shift from a data-poor regime to a data-rich regime, the set of challenges move from the need for new mathematics to squeeze information out of a data set to new engineering, uh, to even handle or process very, very large data sets. Okay. However, some of the methods, some of the models that you'll build are the same in both cases. So database experts, database programmers, and mana and administrators bring a lot of skills to the table to make them appropriate for data science tasks, but there's a there's a focus on a particular data model, which is usually the relational data model. So this is rows and columns. So if you have data coming from sources that are as video or audio or even text or to some extent even graph graphs, nodes and edges, which we'll talk about, a relational database may or may not be the right tool, and even the concepts that transcend any particular database system may or may not be appropriate, and we'll sort of explore when and where it isn't to appropriate as we get into the course. Okay. So visualization experts also bring a lot of skills at the table, but like statisticians are historically less concerned with massive-scale data that spans many hundreds of machines, uh, and then finally, machine learning is perhaps the closest to data science, but here and we'll try to make up more of a point about this later, the the the as a proportion of the time you'll spend on a data science problem actually choosing the the right model or algorithm, machine learning technique, and applying it and running it is a fairly small fraction, which you'll be spending much more time on is the preparation of the data, the manipulation of the data, the cleaning of the data, the wrangling of the data, some have been saying, and for this machine learning techniques are are not particularly relevant. And so this falls back more towards the, uh, the database managers managed the database experts and database programmers. Okay. So there's a lot of courses that could be considered data science courses; some of the new data science and the name some of the newer ones, others have been around for a long time but are but are obviously in this in the same space. And so I want to spend a little bit of time describing the dimensions by which you could describe these courses and then choose a particular point in the design space that we've used to motivate this course. Okay. So as a preface, let me show you this quote from Aaron Kimball, who's a CTO at the CTO at We Do Data. And so he said to me that he worries that the data scientist's role is perhaps like the mythical webmaster of the 90s, that they were expected to do everything, right? The Webmail companies knew they needed to get on the internet in the mid-90s, but they didn't know how, and so they said, well, you know, we'll hire a webmaster, problem solved. All right, the webmaster will write all the content for the website; they'll do the design and manage the user experience; they'll write the code that will wire the website to the order fulfillment system.

In the back end, they'll actually structure the pages and do the navigation. They'll do the logging required to make sure that the site stays up all the time and has reasonably high availability. They'll design the schema to hold the data that will be served out through the website, and so on and so on. It wasn't really feasible that you're going to get this in a single person, and so instead, the internet strategy became a broader team.

Similarly, that might be what we see happening with data science. But here's what it means to me: The term "data science" tells me that if you're a database administrator and your skills are solely about relational databases, the current trend is you will need to learn more about unstructured data and statistical modeling. If you're a statistician, you will need to learn to deal with data that does not fit in memory. If you're a software engineer who's used to sort of building systems and working with files directly, you'll learn you'll need to learn some statistical modeling and how to communicate your results to your managers. You'll need to work with these data sets and actually use them to make decisions. And if you're a business analyst who is trained to make the decisions based on data, you're going to need to start understanding a little bit more about the algorithms and trade-offs, especially at scale.

And for a couple of reasons: One is the costs change dramatically based on the technology you're picking; that's what's happening with cloud computing that we'll talk a little bit about. And what's happening with these algorithms is that you know you can you might be able to get an answer, but it may cost more or less than it did five years ago. The other reason is that, you know, as we do more fly-by-wire business—meaning, you know, we trust algorithms more and more to make some decisions for us—they become these opaque black boxes. And if you don't understand what's going inside going on inside that black box, you're bound to look at the results. So it's not it's no longer safe to just sort of throw your trust over the wall to some algorithm or to or to your staff that's running these algorithms. You may need to sort of understand, internalize the trade-offs, and choosing one model versus another yourself. Okay, so here are the dimensions by which I like to describe these different courses.

The first one is breadth, and so I divide breadth into tools versus abstractions. And so every sophisticated course would prefer to cheat towards abstractions; right, you want you want concepts that transcend any particular implementation. However, what students are interested in is hands-on experience using tools they can use, you know, tomorrow at a job. And so you always have this tension between these two. All right. And so some examples here are, you know, Hadoop, which we'll talk about, is an implementation of an abstraction called MapReduce. And then the app the MapReduce abstraction certainly transcends its particular implementation in in Hadoop. Okay. And so here, as well as I'll mention in the next segment, I want to cheat towards abstraction whenever possible, but make sure that there are assignments; they give you the hands-on skills that people are interested in. All right. The next dimension here is depth. And so by depth, I intend the distinction between structural manipulation of data and statistical manipulation of data. And so here you can think about the relational algebra as a structural formalism, a formalism for manipulating data structurally, while the linear algebra is perhaps a formalism for manipulating data statistically. Okay. And so here, try to strike a balance, but I actually lean more towards structure, and I'm going to defend that position in the next segment.

The next dimension you can think about is scale. And so here is, you know, one end is is yes, it fits in main memory on a single machine versus what I'll call "Cloud," meaning that you might require hundreds of machines to to work on it. And here I cheat towards cloud, and the reasons that I've already sort of described are that, you know, it's no longer safe to assume the data fits in main memory, and to train people to work only with data of that size, you know, the whole world changes when you start moving to even two machines, let alone 100. And to not have give you some exposure to that change would know would not equip you to be an effective data scientist. And then the final dimension I use is sort of the target audience; right, is this for hackers or more for analysts? And by "hacker," I mean, you know, you already have significant programming experience and you're looking to sort of round out your skills in some of the uh mathematics; or are you more of a technology decision-maker who is trying to bring a little bit of technical depth? And here I like to actually strike a balance. I don't want this course to be solely assuming that you that you are a seasoned developer, but nor can nor can we sort of ignore all programming whatsoever. So we're going to try to check a balance between these two. All right. So here's the choices we we sort of made in this course: So one is we cheat towards abstractions; we cheat towards trucks; we definitely like large scale; and then I say we'll strike a balance, but we actually cheat towards the analysts' side. We favor we favor the fact that there are going to be analysts in the room who don't necessarily have significant programming experience. And I've already gotten a lot of questions from folks over email who say, "Hey, look, you know, I haven't done been doing programming day to day; am I going to be able to take something away in this course?" And I think the answer is is yes, although there will be some programming, so so be ready. Thank you. Welcome back to Introduction to Data Science.

So in this segment, I want to talk about these four dimensions that I introduced last time, and I want to justify the first three of them, and we'll talk about this one next time. Okay. And by "justify," what I mean is I want to explain why I've positioned the needles, the uh the the where I, you know, the locations I have for this course. Okay. So the first point here is this dimension of tools versus abstractions, and this may seem sort of obvious that we want to focus on fundamental concepts as opposed to specific tools. But I can appreciate the people that are taking this course and many other courses really want sort of hands-on experience, and we're definitely going to try to strike a balance. But let me try to motivate why I think it's important to sort of focus on this angle, and to do this, let me tell one one particular story that you see happen sort of time and time again. So in this case, we're talking about sort of databases and what is currently going on in the NoSQL systems. All right. So before 2004, you had, you know, the big three relational database vendors plus some open-source solutions like MySQL and PostgreSQL. And then arguably a big event in 2004 was when Jeff Dean and colleagues published this paper on MapReduce at Google. And if you haven't heard of MapReduce, we'll talk about it at length, and if you have, bear with me. So this was great; what this allowed you to do was process very, very large data sets, and it sort of rebooted the database feature set. So it really stuck on it really focused on just scale-out parallelism, and that's it—none of the other features of databases. And this was exciting to a lot of people because they didn't have to sort of deal with the extra features that they didn't need in databases, nor pay the exorbitant license fees associated with a database. So this seemed like, "Boy, this is the right solution." Okay. And you know, a few years it took a few years, but a few years later you had an open-source implementation of the ideas in this paper called Hadoop, led by some folks at Yahoo. Now, even in the same year, one of the earliest and most successful projects within the Hadoop ecosystem was this system called Pig. And what Pig essentially was was a relational algebra programming environment for Hadoop. And if you haven't heard of relational algebra, don't worry, we'll talk about it, but let me convince you that really you notice the word "relational" there. Relational algebra is the secret sauce within relational databases. And so a very early on project that that was deemed necessary in the Hadoop community and was wildly successful was to have relational-style programming on top of this non-relational system. Okay. Moreover, you had other competitive competing projects, uh projects like DryadLINQ, which will Dryad, and then DryadLINQ, which is an interface to Dryad, which also provided a relational algebra-oriented programming environment for large-scale-out parallel data processing applications. Then you had people literally put the language SQL on top of Hadoop. So instead of just the underlying formalism, it literally had the programming language you needed. Then, uh a bit later, you had indexing for Hadoop, which is another feature that databases have that we'll talk about. You had people talk about schemas and more sophisticated types of indexing; it's just two other things the databases have. And then you now start to see transaction processing being a very hot, very important topic in NoSQL systems; is how to support concurrent access at very large scale transparently. And this slide is perhaps a little bit old—it's now 2013, of course, when at the time of this recording—and the Spanner system from Google is an important top important system to look at that we'll talk a little bit about later. Okay.

So now this isn't to say that MapReduce was useless; we're going to talk about that at length, and for a very good reason: It actually has some pretty important permanent contributions, uh three of which I mentioned here. One is, you know, it was the first system to really emphasize fault tolerance. And the idea here, in a nutshell, is that when you're working with a thousand computers at one time for any length of time at all—a few minutes, a few hours—the odds of one of them failing in some way is extremely high. And so databases didn't typically have to worry about this because the assumption, the original, they weren't running on thousands of computers at once. And second of all, they were sort of under the assumption that your queries would typically be pretty fast. And so fault tolerance, what during query processing, so you don't lose all the work you you started on when you're running a query, was something that the Hadoop and, you know, the MapReduce paper really sort of emphasized and has now been sort of accepted by the larger community. The other notion, which is a little more subtle, is this idea of schema and read. And what I mean by that is, you know, the way databases worked in the past and largely still work is, you know, we designed this thing called a schema, which is a particular structure for your data, and then your job is to fit your data into that schema, and until you do so, we don't really want to talk to you; you know, the database has nothing to offer you until you're able to sort of fit into some sort of a schema. Okay. And the observation was that, well, look, a lot of data doesn't come pre-equipped with a schema; we don't have a schema just sort of lying around, and we have to do something with it, you know, and it's huge, you know, it's many hundreds of terabytes or something. So what do we do? Well, you know, one answer isn't you can use these MapReduce-based systems; you know, they do for this, but having to say that before you're allowed to touch your data, you must load it into a database; that was kind of a non-starter for a lot of applications. Okay. And then finally, you know, this idea of user-defined functions is something that all databases or most databases support, and it's the idea that you might want to do things outside of what you can do in a normal SQL query; you might want to write your own code and push it into the database. But the experience of having to sort of write and maintain and manage and use these things is not great, and that's why a lot of people put their logic inside the application as opposed to pushing it down into the database, where arguably it could do more good for reasons we'll talk about. Okay. And so I think, you know, MapReduce sort of argued that, look, you can actually have you can give the Java programmers what they want—a Java programming environment—and let them write scalable systems without forcing them to kind of use this crazy user user-defined function interface that databases offer. Okay. So fine, so what's my point over all this? Well, you know, if we focus too much on tools, what you would get is a snapshot in time of what tools are important as opposed to seeing that some of these features around databases are, you know, they sort of ebb and flow in their popularity, but they're all they're sort of a permanent um they're a permanent value when you're reasoning about large-scale systems. Similarly, you know, you might lose track of what's actually novel and what's actually new uh in in the midst of the conversation about sort of date, you know, relational databases versus versus NoSQL systems. Okay. So I want to focus on these abstractions throughout the course whenever we can. Great. Now. Okay. So fine, we're going to focus on the abstractions. What are the abstractions of data science? Well, it's not clear that people really know yet, and I'll give my case for this in the next slide, but you know the reason why I don't think we really know yet is you see these words being used like "data Jiu Jitsu" and "data wrangling" and "data munging," and this, you know, this is the real skill of data scientists—that they have to be able to wrangle data. Well, what, you know, what does that mean? Okay. So my translation of this is we don't really know what we're talking about yet. But that said, there's probably a few candidates we can consider here. So maybe everything's a matrix, and everything we want to do with data it can be expressed in linear algebra. If you're a database person, maybe everything is a relation, and everything you want to do is expressed in relational algebra. If you're more of an object-oriented programmer, everything's an object, and we communicate between objects by sending messages back and forth through by calling methods. If you're more of a sysadmin type, then, you know, everything's a file, and we write bash scripts to process it. And if you're an R programmer, then maybe everything's a data frame, and we call functions in this library. And MATLAB similarly, perhaps with MATLAB, everything's an array or a matrix or a vector, I guess, and they're in their parlance, and everything's a function on that. Okay. So I think of all these possibilities, there are two that stand out as likely candidates as fundamental abstractions for data science, and those are the first two here. And the reason is is that we see these abstractions appear over and over again independent of particular tools. Now, relations in relational algebra are closely associated with databases, but as I argued a few slides ago and as we'll see throughout the course, you see this come up time and time again. Okay. And you even see it in say, you know, object-oriented languages, and you see it in R, and so on. Okay. So these are the two that we'll mainly focus on in this course. So now I want to motivate desktop scale versus cloud scale. And you know the argument here for desktop scale is that, well, you know, data science is really about the functions and the statistics and the manipulation, the techniques; so therefore, we can sort of push large-scale data into a separate course or a separate category and really just focus on the the the math and the functions. And I think this is a bit of a mistake for a data science course, and the reason is is that, you know, this is a fundamental limitation of a whole category of technologies, and R itself is included in that, although there's a lot of great work on how to sort of scale R up. But as it added, you know, in its basic usage, what you do with R is you read a file, load the whole thing into main memory on one machine, and then call functions on that. And so if your data doesn't fit in main memory on one machine, you're kind of out of luck. Now, you can be clever and start to use indices to kind of limit the data that you need to access, and you can start to try to be parallel to take advantage of the fact that there's now, you know, four and six and eight and twelve cores in your machine in your computers you'll buy nowadays. But trying to be clever and doing that yourself overlooks the fact that a lot of this a lot of these techniques are pretty well understood and already implemented in other systems. Okay. So being able to be cognizant of what other systems can do and take advantage of those flexibly and, you know, write your application in terms of these other systems that already do scale out is a critical skill in data science. And so the point being made in this slide that is somewhat out of date, although you can get the idea, is that simple simple tools that are available in every machine, such as grep—which, if you haven't heard of grep, if you're a Windows user and don't use grep too often, this is essentially search a file for a particular pattern, but it searches it linearly; right, it looks at every single line of the file and checks for the pattern—and so you can do a linear scan of a megabyte in maybe a second and a gigabyte in a minute, and so on. And so at very large-scale data sets, you can't do this linear scan anymore; you have to search in a more in a smarter way. Okay. And you sort of has some cost over here that are probably horribly horribly out of date now. All right. Fine. So the point is large-scale data is not just bigger; it's different. It requires a different way of thinking about techniques and requires a different stack of technologies, and to ignore it's it's a mistake to ignore that in a data science course. Okay. And then this final dimension of sort of hackers versus analysts, and again what I mean here is, you know, do am I going to require sort of deep programming proficiency in order to participate in this data science class and in in general in the data science kind of activities? And I've answered, I don't think so. I think I think we need at least two types of people, and really sort of a broad spectrum of people, and I and this isn't really my idea; this this often-quoted report from the McKinsey Global Institute, you'll see this quote timing again, but but the people that use this quote tend to focus on this first part that talks about 140,000 to 190,000 people with deep analytical skills. But the second part of the quote is, well, you also need 1.5 million managers and analysts who know how to use the analysis of big data to make effective decisions. Okay. So this means that it won't be just the programmers who are working in this space, and I wanted to sort of think about how to design a course that could appeal and inform both categories of people. All right. And this is my last slide of this segment. The other reason why I think hackers versus analysts is that the line between them is kind of blurry nowadays, and technology can actually help here; right, it doesn't require a PhD in computer science or a bachelor's degree in computer science, even in some cases, to manipulate large data sets and for in order to back up this claim, I'll give you an example from some of my work where we have done some work to try to make databases easier to use for say biologists. And this really nasty-looking SQL query that, if you squint closely, you can see that it's actually doing interval arithmetic over genetic sequences; right, this this is a pretty tough query to understand for even experts. This was written by somebody that doesn't do any programming whatsoever; she doesn't write a line of Python; she doesn't write a line of Perl; there's around a lot; right, a line of R, and she's able to write these SQL queries that process very large data sets. Okay. So the fact that, you know, if you understand what's going on and if you can think in terms of some of these abstractions and you understand your problem well enough, you can participate in the activity of manipulating large data sets and doing data science even without a a deep background in software engineering. Okay. And that's why I want to push this needle somewhere over this way; I probably put this in the middle; I suppose I'm not so much trying to focus on only the analysts; I just want to make sure that they're included. Okay. Next time we'll pick up with the last dimension. [Music] Welcome

Back last time, we talked about three out of four of these dimensions in describing how we designed this course in data science. And so in this segment, I want to talk about this last dimension of what I what I call structs versus stats.

And so this is the relative importance of data manipulation versus deeper mathematics. You can see that I've sort of put the the dial here a little bit to the left, and I'll try to motivate that in in the next few minutes. All right. So we already saw one example of this in the first segment where I gave some examples of data science from, you know, recent history. And one of these was Nate Silver's prediction of the Electoral College votes uh for the 2012 U.S. presidential election. And if you recall, you know, this was a con this prediction was accomplished by essentially taking the average of the state polls for each state. Okay, so there it didn't really require a sophisticated statistical model, and yet it had massive impact. Okay.

So, you know, a quote that I think sums us up a little bit comes from Aaron Kimball at a company called Weebie Data where he's he says, you know, "80% of analytics is really just sums and averages." And so if you can get these what do you mean by this is if you get these sums and averages right, if you can do it at any scale on any data that you might see, then you can always sort of build up more and more uh more Advanced Techniques. Everything sort of boils down to just sums and averages. Okay. So I think this is a motivation for why focusing on data manipulation, which can which typically is associated with being able to express sums and averages for example, uh you know what you can do in a database query, which we'll talk about the next couple of lectures, that gets you a pretty long way. Right? You get your 80% of the problem.

So another way of looking at this is that there's three main tasks involved with uh, you know, a data science project. There's, you know, preparing to run the model, running the actual statistical model, and then interpreting the results and communicating it. I got the animation out of order here, so you can ignore that red. But the point here is, you know, again Aaron Kimball from a conversation with him is where I got this uh was the, you know, "80% of work is really in this first step where you're Gathering data and cleaning it and integrating it and restructuring it, transforming it, loading it, and so on." Right? So verifying all these verbs you see you see here, this is this is the hard part, right? And so actually running the model or even or even choosing the model and then running it doesn't tend to keep people up at night in practice. Okay. So and then the joke here is perhaps that the other 80% of the work, you know, implying that there's sort of 160% of a normal task is in data science, is in this interpreting the results. So this is the visualization and the communication and the explanation of the results. Okay.

So there's another reason why I want to focus in this course on data manipulation tasks that are associated with this first number one task. Okay. You know, here another way of looking at this is a quote that's now really old, right? So this is 12 years old or so at the time of this recording from Doug Laney. This is the document that first coined this notion of Big Data being the three Vs of volume, velocity, and variety. And we'll talk about that in a couple of segments, but he has this quote, you know, "No greater barrier to effective data management will exist than the variety of incompatible data formats, non-aligned data structures, and inconsistent data semantics." So this what the database Community, you know, my community calls the data integration problem. This is the hard part. And so he was saying this back in 2001, and I would argue that it's still true today. This is the greatest barrier. And in the context of this, he was talking about this notion of variety being harder than volume or velocity, and I'll explain more about those what those Vs mean in in a couple of segments. All right.

So another vignette here is something that we like to ask the scientists we work with. So these are, you know, astronomers and oceanographers and biologists. We ask them sort of informally how much time they spend quote handling data as opposed to quote doing science. Now, you know, we let them interpret these quotes however they want, but what we mean by doing science, you know, choosing a statistical method or designing a statistical model, they absolutely consider part of their science. And so we mean by handling data is all the other crap, you know, the format conversions and so on. And so what do you think the most common answer is here? Or you can guess to yourself for a second, but they don't even blink. I say things like 90%, and so this number should, you know, give pause. Right? This is taxpayer money that goes to Federal funding agencies to come back to pay some postdoctoral uh fellow to spend 90% of her time doing something that she doesn't even consider science. And so this is why I think it's really important as a data scientist to focus on this problem. Now you might say, "Well, that's just science. What about business?" But uh as I will try to make the point throughout the course, there's a a increasing alignment between what's going on in business and what's going on in science. Okay, and we'll talk about that more in a couple of segments. All right. So if 90% of the problem is handling data, you know, boy, we ought to spend a lot of attention on that.

All right. So, you know, another argument that sort of follows on the first slide that I gave is that structs, you know, the the data manipulation platforms and databases in particular actually go a pretty long way to being able to express more advanced things. And this isn't just a matter of, "Oh, well, you can express anything if you have sums and averages." It's also even uh fairly Advanced Techniques. You can there's an increasing amount of interest in figuring out how to get this stuff into the database. Okay. And so this is a slide I've taken from Christian Grand where he argues that look, you know, if you consider databases versus statistical packages such as SAS or Matlab or R or SPSS, you know, this is what they're doing now. They're taking they're downloading data to use in their favorite statistical package frequently under the assumption that, "Well, of course I have to. Right? Of course that's the only thing that could possibly Express this." Well, look, you know, most of these stat packages the first thing that you'll do is read the data off of disk and load it into memory and then start calling functions on it. Well, if it increasingly data sets simply don't fit in memory on a single machine, certainly not on your laptop. And so you have a couple choices here: either you shift into some kind of fancy cluster version of the tools for which they exist for things like SPSS and Matlab, although they're quite expensive; or uh you sample the data so that you only have only can work with a subset that actually does fit in memory. And you'll see this to be very very common is that it's just it's just a par for the course to to take a sample of the of the data in order to be able to work with it efficiently. Right? But the point here is that this isn't really required if you use different packages. In this case, if you the argument here is that if you can use databases, if you can figure out how to how to perform your tasks in the database, you'll get the scalability for free. Moreover, you know, these toolkits don't have don't necessarily have any kind of notion of parallelism. Right? So even if it does fit in memory, every machine you buy nowadays has, you know, at least four cores and then probably more like eight and soon to be, you know, 12 and 16. So to take advantage of all those cores on your problem um is is something you're going to be looking for in a package, and this is something the databases can do automatically. Most databases, not all. In fact, the the ones you may be familiar with, MySQL and Postgres, typically do not, but other databases will, and we'll talk more about this. Okay. So you get parallelism for free if you can use a database, and you get scalability beyond the size of main memory for free if you can use a database. And that's perhaps a big if, and we'll talk about it.

Okay, let me give you an example here, and you'll actually do this as part of a homework assignment, but, you know, can you express matrix multiplication in this in SQL? And if you can, then, you know, I'd argue, "Well, hey, now any formula that you can express using matrix multiplication you can perhaps express an SQL by doing this over and over again." Okay. And the answer is yes. And in fact, the simplest version of this is is pretty straightforward. So if you haven't ever seen SQL before, don't worry, we'll talk a little bit more about this. But if you have, you know, bear with me. Imagine you have two matrices, A and B. Oops, I'm using the wrong device here. Two matrices, A and B, and what you want to do is find all the, you know, and and the the representation of each Matrix here is as a uh row excuse me, row ID, column ID, and value. Right? So that's your relation. Now this is a very inefficient relation if your Matrix is dense, and I'll let you think about what, well, I'll tell you why, and you can think about a little bit more as well, is that you know an implicit representation of this only has the let's say you have n row, let's say you have five rows and five and six columns, then you only need the 30 values, five times six. But here you're doing you have to do 30 row IDs plus 30 column IDs plus 30 values, so you sort of triple the size of your data relative to, you know, efficient main memory representations. So why would you do that? Well, it turns out that a lot of matrices in practice are sparse, and I put that word right up here at the top. In a sparse Matrix, not all the cells actually have a value, and so you don't actually need to store them. And so this representation in terms of, you know, explicitly having a row ID, a column ID, and a value turns out to be pretty efficient. Okay. And in fact, sparse Matrix solvers, this is exactly the kind of representation they use internally. All right. So if you have a sparse Matrix and if you encode it in if you represent it in a database, then expressing Matrix multiply is not too bad. What you want to do is find all the columns, you know, for each column number in in The Matrix A, find the corresponding row number in column B uh and then add up all the all the contributions to the new value. And I'll show a diagram of this after. In fact, it's like, you know what I'm let me not let me skip going into too much detail about this right now because I'm going to talk about this in detail in preparation for the homework where you'll do this. So right now I guess to take away what I want you to take away is that uh doing, you know, representing matrices inside of a database sounds very unusual. It's actually not the world's worst idea, and in part of the readings from this Mad Skills paper you'll try to you'll see why. So right now I just want you to take away that it can be done and it's not necessarily a terrible idea.

[Music]

Welcome back. So I want to talk a little bit about how the term data science relates to other fields of Science, and in particular I want to introduce this term e-science, which to a first approximation you can think of is equivalent to data science. So while the term e-science is associated with astronomy and oceanography and biology, data science has been adopted more in business, but they involve a lot of the same Concepts. So let me tell you about what's going on in science. So for thousands of years, you know, scientific inquiry has been empirical. Right? You observe the natural world or in some cases maybe replicate the natural world in a controlled environment in the laboratory and make observations about that. In the last, you know, few hundreds of years, science has accepted theoretical models as a valid method of inquiry, you know, one that is reinforcing empirical methods. So, you know, new theories suggest new experiments, and the theories help explain the observed data you get from the experiments. In the last 50 years or so, uh high-speed computation has emitted an entirely new method of scientific inquiry since you can simulate in the computer phenomenon that otherwise couldn't be, you know, you couldn't you can't observe directly and you can't reproduce in the lab, and even the theoretical models uh become too complex to solve analytically using, you know, essentially paper and pencil. Right? But you can actually start from initial conditions and run the simulation to get to get a result. So this is maybe the, you know, what goes on in the interior stars or the shift of tectonic plates or the evolution of the universe or the effects on the ecology from some species dying out and so on. So that's fine. So that's three methods of inquiry. But in the last 10 years or so, there's been arguably a fourth method of scientific inquiry, which is to acquire massive data sets from instruments or from simulations and then explore these data sets using new algorithms and new infrastructure. And so e-science is really about massive and complex data, data large enough to require, you know, automated or semi-automated analysis. You can't look at it; you can't inspect it directly. Okay. And so the relevant tools here are the same as those for data science, you know, databases, visualization, scale-out Computing, maybe even NoSQL systems, machine learning techniques, web services, and so on.

Okay. So the way this this idea of the fourth Paradigm, there's a book that's in the reading list that you can refer to here, and there's a lot of uh there's some other articles in the reading list you can also refer to. The story's been told lots of ways. The way I like to talk about this story is that science has always been about asking questions, but conventionally it was really about querying the world. Right? You would sort of have data acquisition activities, experiments or or field studies that were coupled to very specific hypotheses. Right? You had the question in mind first, and you went out and collect the data. But e-science has really sort of shifted a bit where now you're kind of downloading data en masse; you're downloading the world first, putting into some sort of a representation in the computer, and then querying that database to test your hypotheses. And so it's the data can be acquired independent of any specific hypotheses in some cases. Okay. And this is due in part to the cost of data acquisition being dropping precipitously thanks to Advanced Technology. Right? So the telescopes you can build now that we'll talk about the next couple of slides can acquire at in enormous amounts of uh data at very high resolution. Okay. And in the Life Sciences, you have sort of laboratory Automation and you have high-throughput sequencing. In oceanography, the sensors are getting cheaper; the models, thanks to advances and things to Moore's Law and advances in Computing, the simulations you can run are getting bigger and higher resolution and produce and therefore producing large and large amounts of data and so on. And so, you know, the rate at which data can be produced has far outpaced the rate that we can analyze it or come up with the questions we need to ask about it. Okay. And this suggested a new approach to science.

So let me give you some examples. So we've said that e-science is driven by data more than by the computation. All right. So some examples of the of the size of the data that's coming out: the Apache Point telescope, that was the primary instrument for the Sloan Digital Sky Survey that we might refer to multiple times in this course, produced 80 terabytes of raw image data over a seven-year period. You know, at the time this is a pretty significant data size, and even by many standards is still today. The next generation of this uh the Next Generation project that's in the same sort of spirit of cylindrical Sky survey is the Large Synoptic Survey Telescope. So this guy can produce 40 terabytes per day uh and we'll do so for over a 10-year period. So in total, 100 plus petabytes. And producing the same amount of data that Sloan Digital Sky Survey produced in over its whole entire lifetime, you can produce that over every two days. Okay. And so this is a pretty staggering amount of data and requires a a pretty different approach. One thing I wanted to mention maybe about Sloan Digital Sky Survey: what they actually did here was to take the images uh cook them, right? Extract the relevant objects from it, put all those objects into an off-the-shelf relational database; in fact, it was Microsoft SQL Server; and critically, Host this database online and serve it out over the web. And this required a pretty significant investment in infrastructure, but as a result of doing this, I'm making all the data public and queryable, it became the most productive astronomy facility in history. Right? So the number of papers that have been produced on this data is on the order of thousands, and the original, you know, PIs of the project, the principal investigators of the project, had sort of maybe on the order of 100 papers in mind for the data, and the other 4,900 papers that have been written all came from external Partners writing queries against this database. So just a wild wild success. Now the problem is is that the same technology stack and to some extent even the same approach is difficult to apply in this case of the Large Synoptic Survey Telescope. The reason why this guy's producing so much more data is not just that it's much higher resolution and can perceive a much deeper field in the sky, but also because it's returning to the same point in the sky frequently, every every three days. And so this allows you to look at things that change over time: so asteroids, comments, you might get a supernova and so forth. Okay. And by comparing these images in the time series uh there's all sorts of new questions you can ask. Okay. So both because of the science that they're going to do and because of the sheer scale and because some of the complexity of the details of how the data is acquired, the existing the the previous solution won't work. And so this is motivating a whole new area of research to study data management techniques and data analytic analysis techniques to support this project.

So in Life Sciences, these high-throughput sequencers are capable of producing, you know, terabytes per day when run continuously, and, you know, major Labs that do this work such as the Joint Genome Institute have 25 to 100 of these machines running all the time. Right? So this is spitting out an enormous amount of data, forces, well, I was going to say a variety of samples, so we maybe put individual organisms or it could even be samples from the environment where there's no one particular organism in there, but there's an entire population. All right. So for a variety of uses, these guys are are able to spit out the data. In oceanography, the regional scale nodes of the NSF Ocean Observatories Initiative is a project led here at UW ocean reservation, is a multi-institutional partnership; the regional scale nodes part is run at the University of Washington. So these this project is involved laying, you know, thousands of kilometers of fiber optic cable on the seafloor, connecting thousands of instruments in chemical, physical, and biological uh since thousands of chemical, biological, and physical sensors, including live video from the sea floor to measure to monitor volcanic activity. Okay. So again, the database, the if not a relational database, the the data sets and data infrastructure required to support this effort is significant; it's motivated a lot of new research. All right. All right. In the information space, there's a lot of science to be done directly on the web itself. And so just the web, you know, a single computer can read 30 to 35 megabytes per second from one disk, and so it would take about four months just to read the entire uh web. So new clusters of machines…

Summing up a bit: e-science is about the analysis of data, so the automated or semi-automated extraction of Knowledge from massive volumes of data. And so your main instrument for looking for answers is the are the algorithms and the technology as opposed to direct inspection; there's just too much of it to look at yourself. But it's not just a matter of volume, as we'll talk about in the next segment. This is another link back to what's going on.

In business, right, there's this concept of big data, and there's the three Vs. The Big Data that we'll talk about a little bit more next time, but let me just mention them here: volume, variety, and velocity. You'll read about these, and I'll give you the—the where these—where this terms came from in the next segment. The volume refers to just simply the number of rows or the number of bytes, right, the sheer scale.

Variety is perhaps the number of columns or dimensions, but you know, in science, for example, a lot of—in the Life Sciences in particular—you'll have all right, experiments that involve accessing multiple public databases, as well as multiple sensors, your own data that you've collected and that of your colleagues. The integration task of putting all this data together is a significant bottleneck, even when the mat—even the actual scale of the data is not necessarily all that bad. Okay, so this is the complexity of the data.

And then the velocity—you know, we saw with the Large Synoptic Survey Telescope—that, you know, although this the scale itself is enormous, the fact that 40 terabytes are being collected every two days means that the infrastructure needs to keep up with that pace. And just transferring that data from the telescope facility to the data analysis facility is an engineering challenge. Okay, and you'll also see other Vs here, where things like veracity—you know, can we actually trust this data? So a bit more of that next time.

To summarize here, science is in the midst of a generational shift from a data-poor enterprise, where you can never—you know, there's never enough data—to a data-rich enterprise where there's so much of it you don't know what to do with it. And as a result, you know, data analysis is replaced—data acquisition is the new bottleneck to discovery. Right, so it's not that the cost of going out and getting the data; it's the cost of actually analyzing the data you might already have. So this is fine, but what does this have to do with business, which is probably where a lot of you are—uh, coming from and where your interests lie?

Well, what we see is that business is beginning to look a lot like what's always been happening in science. So they'll—you know, businesses are requiring data aggressively and keeping it around indefinitely in case it becomes useful. They're beginning to hire people that have training and skill sets that look a lot like what's been important in science for a long time, especially mathematical depth, and beginning to make decisions with this data that are very empirical. So we're always wanting to sort of back up every decision with a clear case based on data. And so, for these reasons, I think that you can take the lessons learned in science and apply them in business, and actually vice versa as well. Because one thing where science that's lagging behind business is in the adoption of technology. There's been—there's been proportionally a lot less spent on IT infrastructure in science than there has in business. And so there's this—it's a great time for this—there's this cross-pollination of ideas between both fields. Okay.

And so, you know, going back to this first slide that I gave, e-science and data science have essentially everything in common. So we might use examples interchangeably between the two. Okay [Music] Okay.

So I want to spend a little time on the term big data, and I'm not too concerned with any sort of technical definition of—of the term because it probably doesn't exist. But I want to arm you with some of the language that people use when they describe big data so that you can—you know—speak intelligently about it when—when asked. Okay. So the probably the main thing to recognize is this notion of the three Vs of big data, which are volume, velocity, variety, and we talked a little bit about this in a previous segment. So just to repeat, you know, volume is the size of the data and he measured in bytes or number of rows or number of objects or what have you—sort of the uh vertical dimension of the data. The velocity is—you know, what I'll say here is the latency of data processing relative to the demand for inactivity, and that's maybe a mouthful, but what I mean by that is, you know, how fast is it coming based on how fast it needs to be consumed. And so there's a lot of applications for which interactive response times are—is increasingly important, if not—if not directly important. Okay. And so when this becomes the bottleneck, when this becomes the challenge, then this velocity term starts to become pretty relevant.

And the one that I think is really pretty interesting to me and is near and dear to my hardwood—my research—is this notion of variety. And so here the problem is, you know, an increasing number of different data sources are being applied for any particular task. And so you need to pull out, you know, ASCII files, as well as download data from the web, as well as pull data out of some database, as well as uh—you know—use some NoSQL system, so on. And the integration of all these different data sources is a pretty significant problem and can end up occupying a lot of your time. And so I—you know, I made this point a couple segments ago about researchers who spend, you know, 90% of their time, quote, handling data. This is where a lot of that time is going—this notion of variety. Okay. So all three of these are relevant in performing sort of data science tasks. All right.

Let me give you another notion, and I'm going to go back to use science examples, and you've seen some of these before, but if you sort of make a plot of number of bytes on the y-axis versus number of data sources—maybe columns of data in a single table or columns data across multiple tables or number of distinct data sources on the x-axis—you can sort of map out different fields of study or different problems and sort of see where they lie. And so typically astronomy has been challenged by the sheer volume of data, and so they're sort of up here, where high on the y-axis, but you know—but the number of actual sources in astronomy is not too high. There's telescopes, there's these spectral imagers, and then there's simulations of the—of the—of the Galaxy. And so that's relatively few. In say the ocean sciences and certainly in the Life Sciences, although I only show you know one example here, the variety is really more of a challenge. The actual sheer scale is not as high as the—you know—hundreds of petabytes that can be generated by these—these tele-infrastructures like this—like the Large Synoptic Survey Telescope. Um, but the number of different types of instruments you can use to acquire data is large and ever growing. Right, so you have these glider systems that will go out for months at a time and kind of porpoise through the water. You have autonomous underwater vehicles that are more for short-term mission uh—you know—there's oceanographic cruises where they deploy these conductivity, temperature, and depth instruments that can take profiles of the water. Right, so this is—you know, they had a fixed X, Y, and a varying Z and a varying T—a—they're varying in depth and a varying time, while the gliders are sort of varying in all four dimensions. You have these simulations that are probably one of the largest sources of information, right? So these can be mesoscale, sort of at the order of the—you know—entire hit the Northern Hemisphere or the Old Eastern Pacific, or they could be models of a particular bay or inlet or estuary—um, connected to a river and connected to the open ocean—so much smaller scale thing. So there's a lot of diversity there. And I say stations to mean these sort of fixed stations where there's a particular sensor deployed at one location and just measuring across time. ADCP is an acoustic Doppler something profiler where they're using sound waves to bounce—as to record the time that the sound waves take to bounce off particulate matter in the ocean and again, therefore, measure velocity. And so this gives you an entire profile of the velocities in the ocean. You can mount these on the sea floor pointing upwards; you can mount them on the bottom of a bow pointing downwards, and so on. And then there's satellite images that measure sort of sea color and weed breaching as well. Okay.

So fine. So just a little more on the term Big Data. Uh, the notion that Mike Franklin at the University of Berkeley uses, which I like, is that you know, big data is really relative. Right? It's any data that is expensive to manage and hard to extract value from. So it's not so much about a particular cutoff—you know, what makes it big? Is it a petabyte scale is big versus well, a terabyte scale is small, or a gigabyte sale is very small since it fits in memory on your machine? You know, not necessarily. It depends on what you're trying to do with it, and it depends on what sort of resources and infrastructure you have to bring to bear on the problem. And so, in some sense, difficult data is perhaps a—what Big Data really means. Right? It's not so much about big; it's about being—it's about being challenging. Okay. And so this is really important to remember that big is relative.

So maybe a little bit of history. The term Big Data—there's the earliest notion I could find was from Eric Larson, 1989, where he says, you know, the keepers of—from Harper's Magazine, eventually went into a book—the keepers of Big Data say they do it for consumers' benefit, but data have a way of being used for purposes other than originally intended. So his point was not really about technology at all; it was just a notion that data is being collected for one purpose and being reused for another, which is a theme that I mentioned in the very first segment in this course and we'll come back to over and over again. And so I think he had it right in that sense. So his real point was that—you know—about consumer private data is starting to be commoditized, which was absolutely uh true and fairly prescient at the time since it's become a big issue now. But you know—and it's putting the—perhaps especially impressive that—you know—given that this predates the rise of the internet uh and it already sort of foreshadows very topical issues in Big Data, such as ethics and privacy and sensitivity and so forth that we'll talk a little bit about. Um, but this isn't quite what we mean by Big Data nowadays typically because it didn't have that technology aspect to it; it didn't talk about the challenge of actually managing this—these data sets.

Okay. So another point of reference is that more—more recent reports from some of these consulting firms get credit for these notion of three Vs, but really the original source of this was a report from Gartner in 2001, written by a guy named Doug Laney. And so we talked about volume, velocity, variety, which we've said, but let me just give you a chance to look at these quotes. You know, so in volume, it's—he's really talking about sort of business to business. If you think about 2001, this is around the .com boom, and so everyone's trying to figure out what this new era of technology was going to get—what the internet was really going to give to them beyond just sort of putting up a web page and serving it out to your customers—what—how are you going to be able to interact with your supply chain or your—your vendors and so on. Okay. And so this is what you need by this notion of e-channels, but you know, up to 10x the quantity of data about an individual transaction may be collected. You know, absolutely true that this data exhaust—this point we've made a couple of times—is giving rise to a larger scale of data being collected. You know—and on velocity—well, it's increased the point of interaction speed. Right, so this is that—that need for interactivity uh that didn't used to be so required, but there's the velocity of all business and all transactions sort of increase, so do the constraints on the infrastructure used to process it. And then a variety—you know, I like this one a lot. So through 2003, 2004—right—so he was being sort of fairly conservative about how far—far out you wanted to predict—no greater barrier to effective data management will exist in the variety of incompatible data formats, non-aligned data structures, and inconsistent data semantics. And so this is great; this is—this is—you know, he could have said this for the—through 2015 and then arguably correct. This problem is not gone away. Okay.

All right. So another point in the history of this term Big Data, there was a series of—of talks—a lot of work by John Mashey, who's the chief—who was formerly the chief scientist at SGI, who would talk about Big Data being the next wave of infrastrust. And so what do you mean by infrastructure was what's going to really drive the technology forward? Where are we going to feel the pain? And his point was that the I/O interfaces was where it was tough. So in particular, disk capacities were growing incredibly fast and still are, and that the latencies are not keeping pace. Right, so you can go down to a local store and buy a three-terabyte drive for probably 200 dollars, but the rate at which you can pull data off that is essentially the same as it's been for many, many years. And so now it takes you hours to actually read every byte of data on that disk that you're—that you've stored. And so this is a problem because the actual analysis you can do over all this data—we—you know, we can keep it, and that's fairly cheap, but we can't actually do anything with it because the pipe is so small. Okay. And so this is one of the arguments he made—several sort of uh very coaching arguments about the—where the bottlenecks are in—in the infrastructure, but they all sort of revolved around this idea that—that large—big—you know, big data and big data processing were going to be the uh stress point. And so this is probably a pretty appropriate use of Big Data, although John Mashey has said—you know—that he doesn't—he's not sure that he deserves any credit for uh coining this term since it's—it's a fairly generic term, and he was using it in one way, and we use it now in a related way, but it's not necessarily that it captures everything we need today. And I probably agree with them. All right. So—and just another quote about Big Data sort of today, where we're trying to capture exactly how it is being used—you know, the necessity of grappling with big data and the desirability of unlocking the information hidden within it is now a key theme in all the sciences. And you know, I like this—arguably the key scientific theme of our times. And so we've talked about science; we've talked about the fact that I—I think what's going on in science is more or less equivalent to what's going on in business, and so I think this is—this is nice. Right? It's really maybe the key problem across all fields is unlocking what's going on inside big data—right—getting—getting some—extracting value out of Big Data. All right.

So if I get to the final point I want to make is—you know—where does all this big data come from? And we said this a little bit before, but one is data exhaust from customers. Right, so we're actually tracking a lot more information about interactions with customers than we used to. Right? It's not just about taking their order; it's about monitoring their click stream when that they used in order to get to that order; it's about not just about sending advertisements to them, but it's about watching how many clicks are on each advertisement and whether the—the number of clicks goes up or down depending on where that ad is placed on the web page and so on. Another point that—you know—especially true in science, but I think is also true in business, and I'll give a couple examples of this, is that the availability of new and pervasive sensors—right—we're actually able to get a—get visibility on data sources that we previously couldn't. And I'll give a couple examples in a second. And then I think—you know, I mentioned this already—but the side—the—the technology of data storage—right—just this capacities of disks has gone up; the cost of per storing a byte has gone down. And so we sort of have this ability to keep everything, whether or not we need it, or at least there's a perceived ability to keep everything whether we—not we need it. And so people are doing so—right—the other things that they would have otherwise thrown away, they're starting to keep and then scratching their heads thinking, boy, how might I use this to make predictions and make better business decisions? Okay. Okay.

So just a couple of examples of sensors that may or may not lead to massive data, but just examples of where we're getting visibility on data sources that we didn't previously have. So one is—you know—the fact that all new cars are going to be equipped with these black boxes that are a lot like what's going on inside air—what—what their airliners have, and you know, the reason is for forensics in the event of a crash, but they also record a lot of other information. And so insurance companies have similar devices that you can opt—you can opt in—you can voluntarily plug in to reduce your insurance rates that track your speed—uh, track other kinds of aspects of your driving habits. So this technology—you know—simply would have been hard to imagine, you know, 20, 30 years ago, but now that we have the technology, why not actually start collecting the data from it? Okay. And so while the purpose here is pretty clear, at least for the insurance companies and for these black boxes for crash forensics, you can imagine repurposing that data for other purposes, and in fact, that's what this article is about was—well, boy, you know, is there a privacy risk here given that there's one purpose—you know—they're being deployed for a crash forensics, but they might be used for other purposes. And for insurance companies, you can imagine how they would—are collected with a particular actuarial risk model in mind, but may develop other models in the future given that now they have this data. Okay. So I think this is really a theme of big data is that we're collecting new sources of information independent of how—whether we know we're gonna—how we're going to use them or not in the future. All right. So let me give you another couple of examples from research here by Schwedek Patel and the computer science and engineering department, and there are several related devices, and here I just mentioned two of them: HydroSense and Electrosense. And so both of these devices are intended for consumers to use to monitor their own resource consumption. And so instead of having to kind of rewire your house to monitor your—monitor the consumption of every device in your house, every faucet and every shower and so on, you can just clamp this device on the main water line coming into your house, and it will monitor the pressure changes associated with every individual device. So it can disambiguate that flow and recognize the signature that every device puts on the pressure changes, and so every time you turn on the shower in the upstairs bathroom or flush the toilet in the downstairs bathroom, this thing can tell you—can identify when that happens and give you a log of the events. And by analyzing that log, you can tell—you know—which device—where most of your water is going—is it mostly going for showers? It's mostly going for a dishwasher and so on. Electrosense is very similar; every electrical device in your house puts a recognizable load on the signal—on the power signal that can be read and disambiguated to tell you where—where your energy is going. Okay. So these are just examples of new sensors that are coming on the market where data is being derived that otherwise wouldn't be able to be derived at all. [Music]

Okay. In this segment, I want to talk about the logistics of the course. So how we've organized this course is a guided tour of important trends along with a deep dive into specific topics. And then there's a set of hands-on assignments that are intended to deliver specific skills and experiences, and that's perhaps the most important part. Okay. And so overall, the course is not—you know, the challenge here was to design a course that would be broad enough to cover the topics that we want and also inclusive enough that we didn't sort of have to dial it in for a very specific cohort. But the challenge then is that it's going to be very

Difficult for some people, and others may may find it some some aspects of it certainly routine. I'd be surprised if anybody finds the whole thing routine; if so, then I'd be surprised if you took this course. Okay, so the prerequisites here are pretty light because we are trying to cast such a wide net, but they're really important. So some prior programming experience in some language is going to be really critical. Then you know we're going to use terminology from a basic College statistics or Advanced High School statistics. So when I talk about linear regression, you should know what that means. You should also be able to sort of look at some visualization of data and be able to understand what it's telling you. Okay.

And then perhaps the toughest one is that statistics perhaps the type of one is to have some exposure to databases and databases Concepts. And you know, if you're just starting out in college, that's not always an easy proficiency to have gained or experienced to have gained. Um, but you know, it's not the the we're gonna couch a lot of the discussion in terms of databases and in the relationships to databases, and so some idea of what that means, what they are is going to be helpful. Okay. So to that end, one assignment will involve writing SQL, and if you've never written SQL before but you understand database a little bit, you will probably be able to power through the assignment. If you're an expert in SQL, there's some parts of it that might still be interesting to you.

Having two assignments will be required will involve writing Python; one optional, right, one optional assignment will involve sort of processing big data using Amazon web services. And here, you know, one of the reasons it's optional is that because of the variant skill sets, but another reason is that you'll have to pay out of pocket for the cloud resources. And the reason for that is there's, you know, sixty thousand students who signed up for the course, and we can't sort of pay for all of them. The good news is it will cost sort of less than 10 or so. Okay, and it's optional, so if you don't feel comfortable with that, you don't have to do it; you'll still get full credit for the course. All right.

Then another assignment will involve uh in competing in a kaggle.com project, participating in the cargo.com competition using whatever you want. And so this may or may not involve any programming; a lot of valid assignment, you know, you can you can certainly compete by using Excel and other kinds of GUI tools. Okay, this last bullet probably isn't true, so let's just ignore that actually. Okay, so the learning objectives here is I really want people to come out of this course being able to talk intelligently about the landscape of data science Concepts, tools, algorithms, Technologies. And this will be sort of a springboard for you to dive deeper into particular areas. So for example, machine learning; this is not a machine learning course, but you can but you can dive deeper into machine learning by taking this course. This is not a database course, but you can dive Deeper by taking a database course, and so on. Okay.

Then and I also want to deliver some hands-on experience manipulating data in order to level set people that don't have any programming experience and provide some specific experiences for those of you who do have programming experiences. For example, the first Python assignment will involve competing some sentiment analysis using Twitter data. And so if you already know Python, the learning Python won't be much of a contribution of that assignment, but perhaps it's the first time you've been able to work with the live Twitter stream. Okay, so the end result of this is we hope that you'll be sort of an advanced beginner in a variety of data science topics. And as I said, you know, the tough part here is sort of how to do something more than just superficial access given the data science encompasses such a broad area as we've discussed, and so we think we put together a pretty good program, but you know, you'll have to be the judge of that. Okay. All right, so the risk of belaboring this, the course of the philosophy here has been that the skills needed by the data scientists span a variety of different areas: statistics, programming, databases, distributed systems, visualization. But the traditional organization of these topics is sort of vertical and is not ideal for becoming sort of introductory in data science. Right.

So in order to get introductory level knowledge in all these areas, what you end up having to do is take an introductory course in seven different areas or something, so a lot of different courses. And so our goal is to try to expose and simplify the links between these different areas, okay, as opposed to sort of narrowing our attention on on what makes them unique. Okay, bye. All right, so you know, after taking this course, you will not be an expert in statistics; you will not be an expert in machine learning; certainly you will not emerge an expert in databases and or even no SQL; nor will you sort of have programming efficiency in all these language. However, you will use all these tools; you will understand the basic concepts of all these tools; and you will have applied many of these tools. Okay.

The assignments will there there's a we'll have online short quizzes during the lectures, of which you've already seen some; these finger exercise quizzes. They will be a set of the full length offline assignments as I mentioned, and some of the assignments will be graded by uh some of the programming assignments will be graded automatically; the some of the assignments that don't lend themselves to Auto grading will be assessed using the peer assessment tools. So an example of that is you're going to write up a description of your Kaggle solution in addition to submitting your score for the cargo competition, and other students are going to sort of grade whether whether it's comprehensible or not. So here's my background in one slide, so I have a bachelor's degree in industrial and systems engineering from Georgia Tech, but you know all the problems in industrial engineering tended to be about optimization and automation, which seem to require software, so I sort of got more interested in computer science. So I spent a couple years Consulting with some big firms; Soldier does oil feed Services oil oil field services and Siebel does customer relationship management software, and you probably familiar with Microsoft and Verizon and Deloitte as a managing consulting firm. And then I went back to grad school and got a PhD in computer science from working with oceanographers on query systems for large-scale oceanographic models. And then I spent a couple years working directly with oceanographers as kind of a data architect and before coming to University of Washington where now I lead a group in scalable data analytics or the University of Washington e-science Institute and also an affiliate assistant professor position in computer science engineering. And so there's a bit of a mix of you know, very practical kind of Applied work as well as my research agenda. And so I think that this data science Trend that's occurring sort of strikes close to home with me; I think it's a I think it's a great time for it.

This is a walk through the first assignment for introduction to data science, and here we'll be using Python to process some Twitter data. So the first thing to do is to go down to the link where you can get the class virtual machine and make sure you have that if you're going to use it or to make sure you've installed Python if you are not using the virtual machine. Okay, and I encourage you to read this material about the assignment and about Python, especially if you're new to Python, but I'm going to scroll right down to problem zero. And so some of these steps problem zero included should be pretty straightforward for you if you've used Python, especially if you've used it extensively, but there will be some steps in this assignment that should still give you a challenge. And if you're new to Python, this is a great way to get warmed up. So I've opened the virtual machine here, and there were a couple of things I had to confirm to allow VMware to convert it to the VMware format, but it wasn't too hard. You can also use VirtualBox, which some folks have had done. Okay, so here I am in the home directory, and there's some things that have been automatically exported by VMware from the host machine; you can ignore those, but we have this data side course materials directory, and I'm going to go in to that. And the first thing to do every time you start an assignment is to do a git pull; make sure you have all the most recents changes to the assignment. This allows us to fix bugs in the last minute and not have to worry about how to get you those changes. So I'm already up to date, so I can get started. We can look at what's in I'm going to go to assignment one, which is what this assignment is, and I'm going to scroll down to problem zero and see what it asks me to do. Okay.

So the first thing here is to we're going to access some tweets through the public API that doesn't require any authentication. So this is pretty much identical to just going to the Twitter website and typing in a search term; in this case, Microsoft. And I'm going to do that programmatically by copying this little snippet in. Okay, so we're gonna need a text editor for this, and I I typically wouldn't use a GUI text editor, but I'm going to in this case; I think it might be the most Democratic approach. And this text editor called Gedit is already installs as I'm using here, so I've just pasted this in uh you may or may not be able to paste into a virtual machine without some configuration; I've done that configuration, which amounts to for VMware Mouse to installing VMware tools. I'm not going to go into that here, but hopefully you can do that; if not, then you can type it in, and it shouldn't do too hard; that's really the only thing you're going to have to be able to copy and paste; the rest you'll be typing anyway. Okay, so I'm going to save that; I'm going to assignment one and name it print.py, and now I'm going to go back over to the terminal window, and I'm going to run it. Now I I'm I'm choosing to not use a Python ID development environment; you certainly can if you want. I typically use this kind of a mechanism for a variety of reasons; one is just because it's a little bit more portable when I find myself on a environment that I don't have a lot of control over that and configure; once I get sort of hooked on a development environment, it becomes awkward to not use it, so I sort of prefer to just not be hooked on any of them, but if you have one you like, by all means use it. Okay, so there I just ran print.py, and I see a bunch of junk on the screen, so let's go back to the code and see if we can understand what this junk is. So what this line is doing, this is a response from the website, which is very simple; just a one-liner to open the URL and get the whatever the website sends back to us as a string, and then this library parses that string uh because it under because we're so we're asserting that this is in a format called Json, and this is Json Library knows how to understand that, and so what we get back is a Python object that has all this complicated structure in it. So all this complicated structure was there in the string, and now we parse into a Python complicated object. So what kind of complicated object? So let's say Pi response, and we can check the type of Pi response to understand what we're looking at and print that out, and he says it's addict. Okay, so addict Maps keys to values, and to access the keys in the dict, we can use a method called keys, and I'll do that now. And so this object appears to represent a page of results; there's sort of a page thing that might be the page number; there's an indicator of the next page and so on, but what could the original query you sent, and in fact I'm actually sometimes someone guessing on some of the meaning of some of these; you can read the Twitter API documentation to try to sort this out, or you can just manually inspect it, but this results key sounds intriguing, so let's take a look at that. So how do we actually access the value for a particular key? We use the square bracket oops excuse me; use the square bracket notation and pass the key name as a string, and in Python you can use double quotes or single quotes interchangeably, and so I've used double quotes here. So let's print that guy out. All right, so there's more junk, so the results is some complicated data structures; let's do our standard trick here and see if we can understand what what that is. So I'm going to check the type of this thing, and I'm going to comment this out, and so comments you can use the little hash character, and so that's a list. So what can we do with a list? Well, first thing I'm going to do is put into another variable just to kind of keep myself sane. Let's see these are now the results; print out the first element of the list. And so here we're using the same square bracket notation, but instead of passing a key name, which is what dictionaries support, I'm passing an index number, which is what lists support, and you can look at the Python documentation; there's some links in the assignment to learn about the methods supported by both the dictionary and the list type. Okay, so now we printed this guy; we get one element of the list, which itself looks like yet again another complicated Point that's going to get annoying; another complicated object, and I have you happen to be able to get a feel for this; you can recognize that this is a dictionary because of these curly braces, but we don't necessarily need to guess; we can just check it out once again. So what is the type of this list element? It's a date. All right, same thing; let's look at the keys of it, and notice I'm not caring too much about a lot of software engineering here; we're just trying to inspect some data that we pulled from the web, and this is something that comes up quite a bit I think in these data science tasks is you're looking at some data for the first time, and so it's not necessarily the time to get too obsessive about uh um software engineering either; maybe there may be no one that's going to look at this script besides you; you're just trying to get a feel for things. So there I've printed out the keys for this dictionary, and I see some things I don't understand, but I would argue that perhaps text looks interesting, so let's take a look at the text of the tweet; we're presuming that this is an indeed a tweet. And so now here this is a little bit ugly; you know, I'm not saving the result of the variable; I'm just going to print out directly, but this is okay; this expression let's see; so what the what's the type of this expression? Well, that's a list, and we know with a list we can access it with this square brackets and a number. So what's the type of this expression? Well, that's a dict, and we know that we can access those with these square brackets and a key name, and so the type of this expression is whatever is in text; we haven't actually looked at that yet, so let's do that now. So that looks like a tweet. Okay, and we can try out another one, and that's a different tweet. And if we want to look Loop over all tweets, we can do that too; we can say 4i in use this built-in range function, and now we can print out all the tweets from the first page one one per line, which is actually what the assignment has for now. For this one I'm not don't bother turn anything in; that was just sort of a warm up, especially for folks who haven't really looked at Python before, but that's you know how you can kind of inspect a complicated data structure and try to make sense of it without doing too much work. Okay.

So now we're gonna get access to the actual one percent live stream. So this data was again just like going to the website and typing in a search term and looking at the results, but that's not a we can't get a actual sample of of all the tweets that are coming in, and so for that you need to actually log into your account and do a little bit of configuration, and the steps to do that configuration are here in the assignment, and we'll walk through that real quick right now. Okay, so we go to uh uh by the way here if you are if you're not using the virtual machine, you will need to install the oauth2 library, so make sure you notice that. Okay, so if you first need to create a Twitter account, you don't already have one, and then once you do, you can navigate to this URL, and you'll create a new application. I've got a couple here; you won't have those, and but you can create one with that button. Okay, so I'm going to name this how about assignment one because I name it walkthrough, but why don't we do something more like what you might do, and this is a data science assignment Twitter application, and so what you're doing here is registering an application that's going to consume this Twitter feed; now we're just going to write a couple of scripts with it, but imagine you're doing you're building some sort of a website it was going to do some analysis they want to know about and have a little bit of metadata about who you are, who you're authenticating as, so in case you do something strange they can be aware; I have a little more information about what you're doing with it. Okay, so here you can put in basically any URL, and this is again just a way to help provide some Providence in case you do something untoward; I'm just going to put in my home page, and you can essentially put in anything. Okay, actually yeah, and just to be clear, it actually literally can be anything; it's not even going to be sort of validated. Okay, so fine, so here's and there's a an agreement here and the capsa, and I'm gonna agree to that, type in this guy, and then create the application; oops, that's already been taken, which makes sense, so why don't we say uh data science; it could have been nice complicated name, and I don't care too much about it since we're not actually building the application with this; essentially interesting; I didn't realize this had to be globally unique unless I've already found one. Okay, so now we have some information and some credentials here, but there's one more step; we actually have to authorize this application to be able to use the Twitter data, and I'm not going to go into the details of oauth here; it's somewhat complicated, but necessarily so to sort of support all the requirements we're trying to do to do web-based off authentication, so you can read about it if you're curious, but I'm gonna just sort of blindly follow the instructions in this walkthrough. Okay, so it's been created; it gives me this little green message, but it takes a second to come back, so you're going to sort of stare at this, and nothing's going to happen, and I'm going to refresh, and there it is. Okay, so it's gonna be alarmed if it doesn't immediately come back; it takes a second. Okay, so now we have two sets of credentials here; this thing called a consumer key and a consumer secret and access token and access token secret, and we're just gonna plug these in blindly to this file twitterstream.py, and right at the top of this you've got placeholders here where you can put these, so let me put those in, and we can all right, so there's those, and now the rest of this file is also you need to edit; this is just uh manipulating the oauth protocol to get the results, so I'm just going to save that guy and then run Python Twitter stream, and sure enough, I'm getting tweets from the website, and this will just run forever, constantly streaming down tweets, but we don't want to send it to the screen; we can't do anything with it; we would route this into a file, so I'm going to hit Ctrl C to cancel the stream, and then I'm going to redirect this to an output file, and I happen to know that the format is again this Json format, which stands for JavaScript object notation, and so I'm going to redirect it to that; I'm going to hit return on that, and it'll sit there and silently blink at me forever, so I'm only going to leave it running for maybe a little while.

What I'd ask for in the assignment here is to let it run for 10 minutes or so, so you can get a fairly big data set. There's a step further down in the assignment that we require a reasonable size data set to really be meaningful, and 10 minutes should be enough. And if you want to run it longer, you can. In an early version, just Simon, I actually tried to ask people to run it for um, hours and hours and hours, but that's turned out to be a little bit unwieldy, so I pulled back from that. Okay, so that should be long enough. I'm gonna hit Ctrl C, and I'm going to look at what's going on here. So now I have a four-megabyte file called output.json, and I can look at that file. The first 20 lines of that file with this command, and so that's what you're going to turn in for this assignment is the output of uh, head dots, the first 20 lines of the file. So I'm gonna pick the output of this command and put it into turning.json. You can turn that in on the Coursera website, and I'm not going to walk through that step in the interest of time.

[Music]

Welcome back. I want to talk a little bit about data models as a run-up to talking about databases over the next couple of segments. So this is a data science course, so we know we have data. So the first question to ask is just where is this stored? How do we store data? And so one way of interpreting this question is just to talk about technology, right? So what technology do we use? Well, we use magnetic media, and we might use solid-state drive more recently, right? And so both of these have the property that they're not, they persist even when the power goes off. The data is, the data is safe even when the power is not on. Okay, so non-volatile storage, right? And so a lot of the work in databases is all about how to work with data that's stored on the non-volatile uh, storage media. But another way of interpreting the question is a little different, a little more of the logical way we saw the organization of the data. And so we might ask, you know, one way of interpreting this question is what is the data model we're using? Okay, so it's not just bits on a disk or bits in a file. What do we do? Well, you know, in your personal computer or maybe even your computer, you might store data sort of hierarchically arranged in these kind of nested folders, right? That's one organization of data. And so the data model here is kind of tree-like. Another way is rows and columns, and this is what we'll talk a lot about in this course, right? So in this case, it's a nasty file, and these are hits from a biological database, matches in a biological database for a particular sequence. And of course, you might have spreadsheets that are a little funny, right? Maybe they look a little bit like rows and columns, but maybe they don't. So you know, if you're sort of an embedded table within a spreadsheet and so on. So you need to sort of, the idea is to think about what data model is being applied whenever you think about data, right? So it could be a tree, it could be a table, it could be something a grid like this, unstructured, or it could be a graph, and we'll talk a little bit about that in the future. All right. So in general, what is a data model? Well, there's going to be three components that you should remember. All right. One is that there's going to be some notion of a structure, right? So I've, in the case of tables, it's rows and columns. There's going to be some notion of constraints. What are the legal structures you're allowed to create? So, for example, typically if you think about a tabular data model, you won't have row, all the rows will have the exact same number of columns, right? This is sort of a constraint. You might also have more constraints on the values themselves, such as, you know, this field must be, this column, every value in this column is an integer. And you can even have other kinds of more semantic constraints, such as every column in this or every uh, value in this column must be within a certain range of numbers because it represents, say, days of the year. Okay. And the third one is the operations. And so this is sometimes thought of as independent of these three, but I really like to call the data model all three of these things: the structures, the constraints to, to define valid structures, you know, valid instantiations of these structures, and then the operations you can actually, these start your support. Okay, so let's see some examples. So your structures might be, as we mentioned, your rows and columns, nodes and edges if it's a graphic graph model, key-value pairs has been popular with the NoSQL movement, just a sequence of bytes that might be the structures you have if you're just working with their files. And the constraints you might imagine are, you know, all rows are the same number of columns, as I mentioned, all values and one columns are the same type for a, for a hierarchical view, you might have a child cannot have two parents, right? So this would, you know, for example, one file cannot be in two folders at the same time in your, in your data model of your file system. Okay. And then the operations that are supported, well, maybe you can look up the value given a key X in these key-value pair data models. That's one of the primary operations is if I give you the key, you give me back the value. Uh, for a tabular data model, you might say, well, find me all the rows where one where a particular column has a particular value. In this case, you know, a column last name is equal to the value Jordan. Okay. And with a file, there's not too many operations you can think about. It's essentially get the next in bytes, move to another position within the file, and then you can open and close the file, and that's about, that's not all the operations are supported. Okay. So I think in any case, you see data, especially on non-volatile storage, you can think about what operations are supported, what constraints there are over the structure, and so on, and this gives you an idea of the data model. All right. So what is it? So what is a database? Well, one definition that I think is pretty adequate is, is this one, which is a collection of information organized to afford efficient retrieval, right? So this is a very pretty general definition. It doesn't say anything about tables, doesn't say anything about relations. So when you think about database, don't necessarily assume relational databases. Perfectly adequate to talk about a database that has nothing to do with relations. All right, but it is not just a pile of beta either. Is organized to afford efficient retrieval. All right. So another view of a database is this idea of a schema. And so Jim Gray, in this fourth Paradigm book that I've, that we talked about in the e-science segment of a little bit ago, has this quote: when people use the word database, fundamentally what they're saying is the data should be self-describing, and it should have a schema. That's really all the word database means. And so this goes back to this notion of a data model, right? There's a structure there, and in fact, and some constraints, and that the anthem operations, and all three of these are things you can intuit by looking at the data itself. It needs to be self-describing, right? So if I have, you know, if I have a table, if I have a [Music] a file of data that's organized into rows and columns somewhere, I'm able to inspect which columns it has and how many rows there are and so on, right? I need to be able to understand how to read this data just by looking at the data itself. And so in a database, for example, there'll be a catalog, there will be an explicit schema. So having this idea that the word database is synonymous with self-describing data, you know, data equipped with a schema, that's a pretty common view, and it's an important one to keep in mind when as we go through this course. So let me give you another view of a database motivated by this question: why would I want one in the first place? Why would I want a database? What problem do they solve? Well, this may be four uh, issues you might run into that a, that putting your data into it, into some kind of a database broadly defined can help you solve, and those, these are, these are the ones that I like to talk about. So one is sharing data, right? So you know, once you have multiple users trying to access the same pile of data, some sort of infrastructure or interface to manage concurrent access starts to become required, and this is something that all databases, all I would argue all real databases afford. Okay. Another is, you know, the enforcement of a data model. You might say, well, look, I use a, I use a rows and columns data model, or I use a hierarchical data model, but there's some sort of uh, software that needs to enforce that. So this is something the databases can do. Now remember the data model is not just the, the raw structure of, you know, parents and children, say, with trees or rows and columns. It's also higher level in constraints, such as, you know, this must be a number from one to five, or this must be a day of the week, okay, or it must be one of the customers that already exists in another table. Okay. So these kind of constraints are difficult to enforce in the application layer, and I'll argue this more in a little bit. Okay. So the third reason might be scale, right? So we know we have a pile of data, and we know we have sort of a data model floating around here, but once it gets over a certain size or we get a certain number of instances of this data model uh, we're going to want to use specialized algorithms to be able to work with it. And writing all these specialized algorithms ourselves to traverse larger and larger data sets becomes the bottleneck. And so a database can, can organize these algorithms and expose them through convenient mechanisms. We talk a lot about complexity-hiding interfaces, right? Databases provide a complexity-hiding interface for large-scale data. And I think the fourth one is flexibility, uh, switches, you know, you might have organized, you might have written some software to access your set of files in a particular way, but as soon as you have to access it in some way you didn't anticipate, right? You have to rewrite a bunch of code. And so what databases try to do is anticipate a broad set of, of different ways of accessing the data and working with the data and support all of them. Okay. And so I'm talking pretty abstractly here uh, fairly deliberately because all these things are certainly true of relational databases, which we'll talk about, but I think they ought to be true of anything that sort of earns the term database, and they're certainly true of other systems as well. And so we want to try to think about, keep these things in mind when we think about NoSQL systems and key-value stores in graph databases and so on. Okay. So it's not just relational databases. This is, this is broader than just relational databases. All right. And so I think in general, when you're looking at these different systems and thinking about the data storage layer, which is sort of where we're starting here in this, in this conversation about data science in this course about data science, is how is the data physically organized on disk? You know, what kind of queries, what kinds of operations are going to be efficiently supported by some particular organization, and what kinds are not, right? So one immediate question, well, and this third bullet at least for media question is it hard to update the data or add new data? That's one quick way to split all of, all the different operations you can do on the data. One quick one is the reads versus the writes. And in many cases, you'll find that what's convenient or the organization that makes sense to read the data efficiently is not the same one that makes it efficient to write the data, and that trade-off is at the heart of a lot of the system design challenges in, in databases and other large-scale systems. Okay. But there's other kinds of operations as well, as we discussed, you know, do you want to look up by key? Do you want to look up by some other field and so on, and we'll talk about this in future, in future segments. Okay. Uh, and then what happens if I encounter new queries that I didn't anticipate? Do I need to completely reorganize the data? Do I need to write a bunch of new code? You know, how hard is this? And so this, these are the evaluation criteria you're going to use when you're thinking about choosing a platform for your data science task. Okay, or these are some of the questions, and perhaps not all of them, right? So if broadly your choices are a pile of files, uh, an off-the-shelf database, NoSQL system, or something else, having these questions in mind as you're evaluating the pros and cons is going to be really important. Okay. All right, thank you. So last time we talked about data models and used them to motivate databases and define the term database in, in a broad sense, and now I want to build on that to talk specifically about relational databases. So last time we talked about these questions you could use to reason about different ways of organizing data and evaluate them with respect to your requirements, and I want to talk about these questions and apply them to examples of different kinds of databases you saw in the past uh, that, that ended up motivating uh, the relational model. And in particular, reason I want to go through this is sort of a historical view of things is that you see some of these same designs being proposed in terms of NoSQL systems, and some of the same issues come up, both, both benefits and uh, you know, both pros and cons are still there. So it gives it, it's good to have a historical perspective on them when you're evaluating these modern systems that are becoming popular. Okay. So the questions we talked about were, you know, how is data physically organized on disk? You can ask this about a system. What kind of queries are efficiently supported? How do you update things and so on? All right. So one example is what I'll call a network database, although arguably this is just sort of pre-databases where you just had files, and you, if you go back to our questions, you might ask, well, how is it physically organized on disk? Well, if you, you're using sort of a parts and orders model here, you, you would have an order record, and it would have an address associated with this order record that would physically point to the first part associated with that order, and that part would point to the next one and so on. Another field in the record would point to the customer that made that order. Okay. So going back, you know, what kind of queries are efficiently supported? Well, if I want to find all the parts associated with an order, I can do that pretty efficiently. I have an access to, I have a given order, and I just walk down this chain to gather up all the parts. What kind of queries are not efficient supported efficiently is, you know, I want to find all the orders that uh, involve a particular part, although all the orders that involve this washer, well, now I have to scan every order to look for them. Okay. There are some ways around that by putting back pointers and so on. Another problem with this file-oriented, you know, sort of proto-database model is that whenever I want to make a change to the data, right? If I want to have an extra field added to support the billing customer as opposed to the shipping customer, well, I've just added a new field, I've extended the length of this record, that means that everything else below that record needs to be moved. More importantly, all the programs that, that navigate this structure now need to be aware of this other field. They all need to be rewritten to accommodate this extra piece of data. Okay. Moreover, if you, if you want to support different access methods, as we talked about, if I want to look, say, by part and find all the orders, I end up having to make a complete second copy of the database. And now when I update, when I make a change to one copy, I need to make a change to all copies, and you can imagine how the space of possible copies might grow might get pretty big. Okay. So a partial solution to this problem was this notion of hierarchical databases characterized by perhaps IBM's IMS system, which actually still exists and still has customers. And so here the idea was to order organize data in terms of segments, but still, the logical model was they had this hierarchical flavor that we saw in the network model as well. So here I've switched, I've made the top-level access the customer instead of order, and so logically what you have is that all an order is only located underneath the customer, and a part is only located underneath an order. However, given that they're in separate segments, I can make a change to one segment without having to break all the code that accesses other segments. The downside that still exists here though is that the programmer, the application developer still needs to understand this hierarchy in order to find anything. Okay. They have to actually know exactly how things are organized, for example, that orders appear under customers. So you still have to anticipate what kind of access methods your customers are going to want and design for those. All right. Updates here are a little bit easier given that I can add an order to one segment stored elsewhere without affecting all the other structures, and I can even add a field, and I can only, only make changes to the orders as opposed to changing everything. Moreover, the software layers on top of, of this were able to sort of insulate from those kind of changes with us with some reliability. Okay. So this new field would only be passed back to the client when they actually needed that new field. Okay. So there's some measure of what I'll call data independence, and we'll talk about that a little bit more in a few minutes. Okay. So moving towards relational databases, the one view of what a relational database really is is here I'm quoting Kurt Monash, who's a, a an analyst for the database industry, and he says, you know, relational database management systems were invented to let you use one set of data in multiple ways, including ways that were unforeseen at the time the database was built and the end at the time the first applications were written. And so I want to emphasize here is that this is the key idea of relational databases, not, you know, SQL and not some of the other things you may associate with, with particular implementations. It's really just about organizing the data in such a way just to support unforeseen access methods, querying in ways that you didn't anticipate when you organized it, insulating applications from changes. Okay. So what is a relational database? Well, at the simplest level, everything is a relation, which is synonymous with a table, right? Everything's rows and columns, and this probably doesn't need to be made explicitly, but let me do so. Every row in the table has exactly the same columns, has the same number of columns, but they also have the same types. Okay. So if a column is an integer in one row, then it needs to be an integer in all the rows. All right. And then a consequence of this model of everything being a table means that you don't have pointers anymore, right? You don't have physical addresses. All you have is tables. And so relationships between different data items are implicit. So instead of having the, so here we've switched to the, the domain to one of courses and students. So this table is a student takes, we'll say takes a student takes course, and this is a student record. Well, instead of having a physical pointer from the course record back to the student, we just have a shared ID. The only, the only relationship between these, these two data items is the fact that they both have the same value in a particular column. Okay. And so this is, you know, intuitively this sounds really bad for performance, right? Off the bat, right? If I want to go look up all the students associated, all the students' names associated with a particular course, once I have my course, I need to go look up in this table all the values that match as opposed to just navigating directly to them, which you could do with the hierarchical method. But if I want to go the other direction

It's the exact same process. I look up the name I want to find; you know, all the courses that a student, student has taken, right? I can do so the same way. I do have to do the lookups, which maybe is a cost and performance, but the mechanism by which I look things up is the same in both cases. Okay.

Moreover, everything is stored only once, which is which is a feature of the hierarchical databases we're able to achieve as well in most cases. Okay. And, but the network databases were not; we don't have multiple copies of things lying around. All right.

So the philosophy here is, you know, being cute about this, you know, the quote from the 19th century is that, you know, God made the integers and all else is the work is man. Well, you know, Todd made the relations, which is a reference to Edgar Codd, who wrote the first relational database paper and went on to win the Turing Award for his work, which is sort of the Nobel Prize in computer science. Codd made relations and all else's work is man. So everything is a table is the number one thing to remember about the relational data model; everything's a relation. All right. So let's actually break here, and I'll pick up with this slide next time. [Music]

Okay, so let's talk about relational databases. So the history here is that, which I motivated last time I hope, is that pre-relational, if your data changed in some significant way, if you need to reorganize things in some way, your application broke. Okay. So if you changed the parent-child relationships in the hierarchical model, or if you pretty much did anything with the network or file-oriented model, you, in your applications had to be rewritten to support that. Okay.

And so early relational databases address this issue, and even though they were buggy and sort of slow, they required only about five percent of the code you had to write previously. And so this was an enormous win. Okay.

And so this quote sort of motivating, uh, sort of following on what the quote I used from Kurt Monash in the previous segment is from the original paper on databases from, uh, Ted Codd: "Activities of users at terminals and most application programs should remain unaffected when the internal representation of data is changed, and even, excuse me, when even when some aspects of the external representation are changed." And so the reason I want to emphasize this is that again, this is the key idea of relational databases, not SQL and not some of the other features that you associate with particular implementations; it's really this notion of data independence. Okay.

And this was right there in the abstract in the original paper; this is the key idea. And then, and the reason I'm hitting this so hard is that this idea is still just as important now as it was then. All right.

So I want to go through some of the other key ideas that were, uh, that are associated with relational databases, whether or not they were in the original paper. So one key idea is that programs that manipulate a tabular, the manipulative tabular data exhibit this algebraic structure that we can use to reason about them and manipulate the logical model independent of any physical data representation. So what I mean here is that if you think in terms of tables, and you think about the operations the table support, you can think about what your program means and even how to optimize it, which will show regardless of how the bits are actually organized on disk. And this is, you know, incredibly powerful. Okay.

So the key idea here again is physical data independence, and we'll talk about what logical data dependence means too, in the next segment. And so, you know, the programs that you write to manipulate things no longer have to sort of manipulate files and sort of chase pointers around; you can, in this case, access it through a high-level language, SQL, although again it doesn't have to be SQL; the point is that you're manipulating logical structures called tables. All right. So just know the term physical data independence, and know that it means that your programs you write to manipulate data are more robust than they would be without this relational model. All right.

So another key idea is that there's this algebra of tables that I mentioned, and we'll talk more about these specific operators in a bit, but at a high level, you know, one operation on the table is to select out rows that satisfy some condition; another is to ignore columns that you're not interested in; another one is to, for two tables, for every record in the first table, find corresponding records in another table; right, select, project, and join, and there's other operations you can define, as to aggregation, all sorts of set operations derived from set theory, union and difference, in cross product, and so on. And so these operations, if you write your expression out in terms of these operations, it's very clear what it means, and it, it's, it's, for software engineering purposes, it allows the database designers to focus on just implementing these operations efficiently. Okay.

Now, yeah, if I could, if I had you in the classroom, what I'd ask is how many people have heard of the relational algebra, and also asked how many people have worked with databases, and typically the number of people who've worked with databases is very high, and the number of people who have heard of the relational algebra are somewhat lower, and that's one of the things I hope to fix in this course is to equate the two. Right, if you, if you, if you understand databases, I want you to understand relational algebra, and, uh, vice versa, I guess, comes for free. Okay.

So why do we care about this algebra? Why am I saying algebra? Well, you know, what I, when, when I'm giving a talk and describing using this slide, what I'll also ask is how many people have heard of algebraic optimization, and typically very few have, even if they're computer scientists, unless it's a room full of database people. But the thing is that you already understand what this is, right? You don't have to know databases to know what this is; this is just something you learned in high school in algebra class. Okay.

So forget tables for a second; just think about integers. Well, I've got this expression here, and I want you to evaluate this expression when I tell you Z is equal to four. Okay. So one thing you might do is just, well, say, well, you know, 4 times 2 is 8, and 4 times 3 is 12, and so that's 20, and I have zero, and that doesn't change anything, and then I divide by one, fine. But if you're clever, you might notice that, well, adding 0 to any number doesn't change it at all, so I'll just ignore that altogether. Similarly, dividing any number by one, or any integer by one, is the same number, so I'll ignore that as well. And then, if you're really clever, you might notice that there's a distributivity law here that says when I see this pattern, I can pull out the multiplication, and by applying these rules in turn, including commutative laws that allows things to be reordered, I can simplify this expression down to this, and this just says, well, now if 2 plus 3 is 5, just multiply 5 times 4, and I get 20. And I've done few operations; I've only done two operations instead of five, and I didn't have to do division, which is potentially an expensive operator if you're thinking about a computer evaluating this. Now, do you know, do computers use this kind of symbolic reasoning when they evaluate expressions, uh, in over integers? No, the answer is no, and the reason is is that this kind of symbolic reasoning is much, much more expensive than just from evaluating the damn thing. Right. So fine, but if, if the objects that you're manipulating are not small integers, but rather, you know, terabyte-size tables, then this kind of symbolic reasoning is not only, uh, valuable, but it's absolutely critical. If you do things in the wrong order, if you do wasted work, or you do more operations than you need to over massive tables, you're dead in the water, and you'll get nothing done. And so all databases, all relational databases rather, do this kind of algebraic optimization when you write a query, right?

And so if you think in terms of SQL, if you're familiar with SQL, your query gets translated into a relational algebra expression in terms of selects and projection joins, and then is manipulated according to algebraic and rewrite rules just like you learned in algebra class, and that's why the term algebra is there, uh, and they attempt to simplify the expression, and I simplify that the reason I pause it simplifies perhaps not the right word because it's not, it's not always true that the shorter the expression, the faster it is; it's, we actually use this notion of cost-based optimization, which means we'll try lots of different kinds of equivalent expressions, assign each one of them an estimated cost, and choose the one with the lowest cost. And this is something that all relational databases are doing in one form or another. Okay.

So fine, so this is this is the magic trick of query processing in relational databases, and this is a really, really great idea, and the reason why this works is because we understand very formally what these operations are and what they mean. Okay. And so when you, when you relax this formal model and start allowing anybody to write any kind of code they want over the data, you lose the ability to do this kind of algebraic optimization, and you leave it up to the programmer to write the best possible algorithm. And what I'm hinting at here is, we'll talk about it more later, but when you think about writing large-scale data processing pipelines in something like MapReduce, and if you haven't heard of MapReduce, don't worry, we'll talk about it, you're leaving all the work up to the programmer to not only write the logic but also to do the optimization, and this, you can pay, this can impose a penalty. Okay.

One final comment about this is, is the term algebra is not just kind of trying to connote, you know, algebra from high school; it literally is the same thing. And so what, when you hear the word algebra, what you should be thinking of is this notion of algebraic closure, and what I mean by that is every operation that applies to a table also returns a table. And so I can chain these operations together to, um, always get tables, right? Now that's the exact same idea that's going on when you talk about operations over integers or, or real numbers, and you might sort of quibble and say, well, if I divide an integer by some other integer, I may get a real number, and that's true, but there's notions of multi-sorted algebra with different types involved, but the point is that this notion of closure, you can't escape the system by applying operations, is always true when you hear the term algebra. Okay. So we're not making things up. Fine.

So here's some relational algebra expressions that, if you squint hard enough, you can see kind of look like similar expressions over integers, except instead of addition and multiplication, we have things like joins and selects. And so what this says, I haven't shown you the query; I don't expect you to necessarily see this, but what this says is, select certain values from a relation R, and here I'm going to select other values from the same relation R; that's okay; I can have two different, you know, the relational R can appear in two different places in the same expression, no problem, and then join them together; then select still other values from the relation R and join this one; and one way of evaluating this plan is to perform this join first and then this joins second, indicated by these parentheses, right? Uh, another way to evaluate this expression is to perform this join first and then form this join second, indicated by the parentheses; still another expression is to take the full cross product of all three relations, which I haven't told you across product here; this is actually a pretty bad one to do; if you do know what a cross product is, it generates an enormous amount of data, and you would never actually want to evaluate this plan, but you could, and it's provably equivalent to these other plans, so you know that it returns the same answer, and now all we have to do is figure out which one of these is likely to be the cheapest one, and the, we'll choose that one to run. And this is something that this is the kind of reasoning that all databases do internally whenever you write a query. [Music]

Okay, so where are we now? So we've given an overview of data science itself, and one of the things we talked about was that there's this important aspect of, you know, data munging or manipulation, cleaning, restructuring, and so on, that is perhaps, you know, ill-defined, but is kind of what keeps people up at night when they're working on data science problems. Okay.

And we also gave an overview of relational databases, uh, kind of a history of relational databases and why they came into being in the first place, and, you know, we found that the original problem being addressed was this one of physical data independence, that, you know, when aspects of the data changed, all the applications broke, and so you wanted to insulate application from certain kinds of changes. Okay.

And one of the tricks here, the secret sauce of relational databases, is this algebra of tables that allows you to reason about manipulation tasks, reason about data manipulation tasks independently of the grubby details of the physical representation. Okay. So this idea will come up over and over and over again, even outside of the context of, say, you know, Oracle and Microsoft SQL Server and IBM DB2 and so on; you know, you don't have to be talking about a commercial flagship relational database system to make use of this relational algebra. Okay. And we'll see that. And so I want to spend some time in this segment and then probably the next few segments on understanding the relational algebra, and so, you know, at times this may look like more of a theoretical exercise, but I promise you it's not, right; these, there's, there's an entire database course offered, say, here at the University of Washington and everywhere else that I think is a great idea to take; we're taking out segments of that that are demonstrably practical in a data science setting. Okay.

So I'll also mention that, uh, many of the slides in the next segment or two came from the Introduction to Data Management course developed by Dan's teacher, Magda Balzinska, and is taught here at the University of Washington. Okay.

So the relational algebra operators that we hinted at, but maybe not listed out explicitly, are these: include the set operations that are lifted to support relations, and we'll see examples of that, and then the big three are selection, projection, and join, right? It's just, and we'll talk about the meaning of those, and then there's these extended relational algebra operators that have to do with manipulating duplicates of tuples, and on the next slide I'm going to explain where duplicates come up and why it's important to make a different distinction between working when they're in the presence of duplicates and when they're already duplicates. Okay. And these include just an operator to eliminate duplicates altogether; there's a group by operation you may be familiar with if you worked with SQL, and this operator appears here, and then you can sort and so forth. These are extended in some, these are also extended in the sense that I shouldn't say just duplicates; it's also sorting, for example, doesn't deal with duplicates; it's extensions in relational algebra off of, you know, away from the pure set-based, set theory-based model. So, for example, a set of objects doesn't have any kind of order applied to it, and yet we were allowed to sort things in SQL. Okay. And it's a practical, you know, it's something that's that's practical for applications to be able to define what kind of order the tuples come back in, but it's not part of the formalism; it was added in afterwards. And so that's this extension between the pure relational algebra and the extended relational algebra. This is probably as close to the theoretical underpinnings that I'm going to care to get; the difference between these two comes up a lot when you're trying to prove properties about, uh, the formalism, right? Because the extended relational algebra is much more, much more difficult to, uh, prove things about, if you can at all, but as a practical matter, the difference between these two classes of operators is not particularly important. Okay.

So the takeaway here is that there's a big set of a meat, you know, a rich set of operators, but if someone says the relational algebra, see the first thing you should think of is set operations plus selection, projection, and join. Okay. All right.

So this notion of sets versus bags, the duplicate question. Well, so first of all, what is a set? A set is a collection of objects where there are no duplicates, and a bag is a collection of objects where there can be duplicates. And so, right up here, you know, A is repeated, is not repeated at all in a set, but it may be repeated in a bag, and whether that's legal or illegal is what gives you the semantics of a set versus bag. So you can define the relational algebra in terms of these two different semantics; you can define in terms of set, or you can define in terms of bag, and this notion of an extended relational algebra comes from the need to sort of work with bags as well as other things like sorting as I mentioned. Okay.

And so the rule of thumb here, this last one I'm really going to mention this, the rule of thumb here is that every paper you read, if you're, if you're, if you end up reading some of the papers if you talk about in this course or beyond, will, you know, unless it's set explicitly, we'll assume set semantics. Okay. So be prepared for that, while every implementation, you know, every commercial database will assume bag semantics, and we'll sort of see where that comes up in the language. Okay. So I just want to put that out there up front that, you know, we may play fast and loose with the difference between sets versus bags, but it can be important in practice. Okay.

So one lifted set operation, you can define the union of two sets in the standard way; the union of two relations is natural given that a relation is a set of tuples, and in relational algebra notation, I'd write it like this, and I can also write it in SQL with the UNION keyword, and here's where set and bag will come up; if I want to, by unqualified UNION does indeed remove duplicates, in which case the answer is of the really the union of this relation with A1 B1 as a tuple and A2 B1 and A1 B1 and A3 B4 is these three tuples; the duplicate of A1B1 didn't get passed through; to express this in bag semantics, to make sure we do include duplicates, you can say UNION ALL, and that would include all four tuples. Okay.

You can define the difference operation, uh, the same way, or in the same way in the sense that you're lifting it from the set from the natural definition of over sets that find every, every, uh, tuple in this set and remove any tuples that also appear in this set, and we see one, we see A1 B1 as we saw before also appears in R1, and so you take, you get rid of it, and all you're left with is this tuple. All right. So why isn't this one in there? Well, we don't, if it doesn't, if A3 B4 doesn't appear in R1, we know it's not in the set; all we want is everything that's in R1 removing things that also appear in R2. Okay. All right. So what about intersection? That's another set operation that we could lift up; you can indeed define intersection, but you don't necessarily need to have it as a fundamental operator because you can re-express it in terms of difference. Right. So if I want the intersection of R1 and R2, I can take everything in R1 that is not in R2, and then I can take everything in R1 that is not in that result. So if you think about this for a second, this expression returns everything that isn't that is only in R1, and then this expression overall removes everything that is only in R1, leaving things that are both in R1 and R2, and so that's what intersection is. Okay. And we'll touch on this later, but you can also express intersection in terms of join, which that operator we haven't defined yet. Okay.

The selection operator is how we take tuples that satisfy a certain condition, and so we write it with the, with sigma, and we put C to express the condition. This notation, honestly, we won't necessarily use too much throughout this course, but I think it's good to be familiar with it when it does come up; I'm more interested in recognizing the sort of English translation of these, select, union, join, and so on; the Greek notation is probably is

Perhaps less important. Okay, so if we want to find where the salary is greater than 4,000 over an employee, or where the name is equal to Smith, that's that's an instance of a selection operator. And the let's see, it says the condition C can involve equals, you know, less than, greater than, equal to, and so on, but it can be more than this, right? It can be any sort of Boolean expression. And in fact, as we'll see in maybe a segment or two, it it can be sort of any arbitrary function that returns a Boolean value. So it doesn't have to necessarily just be our a less than b or a equals B; it could be some complicated function. But let's say in between some complicated function that's user-defined and a simple condition like this where you just say salary equals four thousand, or sorry, salary greater than four thousand, you can have arbitrary Boolean Expressions. So you could have conjunctions; you could say where salary is greater than four thousand and yes, name equals Smith. And of course, you can say or, and you can say not. All of these are legal.

Okay, so as an example, if we want to have a selection where salary is greater than four thousand of an employee, which excuse me, which one pass this test? Well, I've been saying four thousand this whole time, and that's forty thousand. Excuse me, these numbers looked for a moment. So uh, John has a salary less than 40,000, and so he can be removed from the set. And so the results of this expression is this table. Right, we have tables in and tables out. The result of this expression is this table, same three columns and only two tuples in it. Okay, I guess sometimes I gesture here, and I'm not sure you can see it when I write out. Maybe you can.

Okay, a project operator uh eliminates columns. And this is another one where you have to be sort of careful about set versus bag. So when you see a projection in the set semantics, you're going to remove all columns that aren't explicitly listed, but you're also going to remove all duplicates that might remain. So if I project away all columns except for the last name, then you might think I get, you know, Bob Smith and John Smith, and I get rid of first name. I'm left with two tuples or one Tuple, right? Well, in set semantics, everything must be a set, and so duplicates get removed automatically. In bag semantics, they'll both be there. And if you write this query in SQL in any commercial database, what you'll get back is two two tools, both Smiths; you'll get duplicates, and then you can explicitly ask them to be removed with a different keyword called distinct. Okay, and we'll see an example of this. All right, so fine. So as an example, here is Project social security number and names. We only want two columns out of all the original columns that might appear in the employee table. All right, and so here's an example of the set semantics versus bag semantics. Here's the original table with three columns, and we project onto name and salary, and so that means we get rid of SSN. Well, now we have three instances of a guy named John and with three different with uh three salaries. And the bag semantics would be okay. Without the set semantics would remove these uh would remove one instance of these of this duplicates. And so the question here is which one's more efficient? Well, removing duplicates is expensive, so just leaving them in place is is more efficient. And in fact, that's I I that's really the motivation as to why um commercial one of the key motivations why commercial databases assume bag semantics is that on every single query where you sort where you want to choose which columns you have, having an extra step to remove duplicates is sort of wasteful. In typical applications, you may not care, or they may you may sort of know by domain semantics that the remaining columns are unique anyway. Okay, so you can tolerate duplicates, and so forcibly removing them just to be pure with respect to the underlying semantics of the uh formalism was not particularly necessary.

Okay. Okay, so an operator that you won't hear much about, but I do want to mention for a couple of reasons: one is that it's it's a useful as a reasoning tool when you're thinking about manipulating tables, and then the other one is it actually is coming up more and more often in sort of analytics applications and data science applications. It didn't come out very often in traditional relational database applications, and that's this notion of cross product. And so a cross product here is for every Tuple in R1 uh for every combination of tuples in R1 and R2, produce a tuple in the output. And so the size of this thing is the size of R1 multiplied by the size of R2 equals here. I'll just do it this way. Oops, excuse me. All right. Okay, so fine. So an example where you might see this and why, you know, sort of give you a flavor why this is coming up more and more often is that you often want to find all pairs of objects that have some similarity condition. And while there are while in any particular instance of this problem there might be tricks to do this more efficiently, the brute force method of just generate all possible Pairs and then apply some similar to apply some function to determine their similarity is often used in practice. Okay, so for example, if you want to compare two images for similarity, right, you have two faces and you're trying to see if they're the same face, and you've got a piece of code that can do this uh you now need to run you and you have two big two big tables full of uh images or just collections of images. Well, generating all possible Pairs and applying the function is always a reasonable way to do this. Okay. All right, so these kinds of operations are coming up more and more often, and I I'm pausing because I was wondering whether to sort of mentioned briefly some of the techniques you can do to get around this, but I don't think it's going to come through very well without a little bit more visual aid. So let me let me skip that for now. Okay. All right, so what does the cross product look like? Well, imagine you had a table employee with these two columns on a table dependent uh dependent with these two columns, and we do a whoops there's a like there. We do a cross product of employee with dependent. That's all possible combinations of employees with all possible combinations dependent. So we know there's going to be four tuples in the output, and you can check to see that John is here twice, once for every instance and dependent, and so on. All right, so now let's talk about join, and I probably maybe should have put a slide in here just about join in general first. So really, we're talking about join the most common instance of join that you're going to run into, and in fact, if we don't qualify it, this is what we mean is is what I'll call equa join. And so ecuadillion is a join within a quality condition, right here. Okay, so what does join do? Join says for every record in R1, find me a corresponding record in R2 that satisfies some condition. And in general, especially for those of you familiar with databases, you're going to be thinking about in terms of equity join. This would be, you know, for every course in student if find these student IDs that appear in particular courses. Okay, so join John primary key and foreign key, if you don't mind the jargon, are our instances of ecwid joins. They they need not be. Is one point I want to make. So we're not talking too much about schemas in this course. I'm more interested in teaching about the relational algebra and showing how it comes up rather than teaching about how to design a database in the first place. And the reason for that is that you know you often don't have the luxury I made this point before. We don't have the luxury of a engineered schema. You're not at you don't have time to build one. You don't have uh there isn't one handed to in the first place. There perhaps isn't even much need for one in that you're going to get your answer to a couple of questions, and then you might sort of move on to something else. So there's not really a way to amortize the cost of developing the scheme in the first place, and so on. So you know, if you've worked with databases, most of the joins you're doing are going to be along these predefined relationships that are called foreign keys, but they need not those need not exist in order to apply a join. Okay. And then I'll point out this is just just a sort of syntactic note. You can write this two different ways in SQL. You'll sometimes see select star from R1 join R2 on some joint condition, and other times you'll just say select star from R1, R2 where this condition is met. Now, if I literally translated this into a brain-dead relational algebra plan, and we'll talk about sort of how to do this a little bit more mechanically in a bit, this is actually sort of saying, well, hey, look, first build the cross product of R1 and R2, right, and then filter that cross product such that this condition is met. Okay, and this one is saying, no, no, don't do that. Actually, you know, use the join operator uh rather than generating the cross product. But the optimizer, the databases are not that stupid, right? They're smart enough to figure out that even in this case, the right way to do this is to express a join. And so in practice, there's no difference between these two different ways of spelling the same the same query. Okay, and in fact, you know, to a first approximation, two equivalent queries but different different syntax but same semantics in in SQL, there's not going to be any difference between them. The optimizer doesn't care how you write your SQL; it's going to optimize the thing anyway; it's going to turn it into a relation algebra plan and manipulate that plan to find to guess the best way to evaluate that query. Okay, and so I say to a first approximation because there are such things as query hints and other ways you can sort of uh tell to tell the optimizer what you how you'd like the query to be evaluated. We're not going to talk about that because they're rarely important, and you know the the other thing, it's not impossible to have two different queries that do the same thing that the optimizer can't figure out are actually equivalent, and so it's not impossible to get two different plans, but typically that won't be the case, and in this example that doesn't matter at all. Join versus versus in the where Clause, I'll tend to write queries this way if and when they come up. Okay, fine. So that's the most common and the simplest instance of this join operator. Okay, in fact, I think I did as a disservice here by doing this on left and right. This is actually kind of a nice example. This is the SQL equivalent to writing join, and this is the relational sort of the SQL equivalent of writing cross product followed by a selection, right? They're the same. In fact, the only reason why in the algebra, you know, remember the algebra as a formalism, you know, maybe you don't care so much about deriving new operators as long as you can express the thing you don't it doesn't matter. There there's so much work, and there's and there's so many good algorithms for implementing join that it sort of deserves its place as a specific operation. We don't want to have to write cross product followed by a select when we're actually sort of talking about the efficient joint algorithms.

Okay. All right, so more generally, you can have what we'll call a Theta join, and this is essentially just a join, but the condition here can be anything you want. Okay, rather than just an equality condition, this could be greater than or less than or arbitrary functions and so on. And so there's all pair of similarity test that I talked about before is an example of um a Theta join, and we'll see a more detailed example in a second. And so just to point out that that equidjoin itself is a special case of theta join where where Theta is just the equality condition. All right, so there's some examples of theta joins just to sort of demonstrate that these come up pretty often in practice, more than you might be familiar with. And again, especially speaking to the people who are familiar who have experience with databases uh you know, these are not going to be a long foreign key relationships quite as often, right? Okay, so if you want to say find all hospitals within five miles of a school, well, you know, this doesn't immediately seem like a relational algebra query or a SQL query, but it kind of is, right? It's just a join where the join condition is this distance function over the location of the hospital and the location of the school. Okay, and then I did a projection here to sort of project out the name of the hospital because the the English version of this seemed to suggest that we just want the name of the hospital, and that's it. Okay, and so in SQL, this would this might look like this where you say, look, give me all combinations of hospitals and schools, and then filter on the ones where the location of the hospital is less than five miles away from the location of the school. And here I'm kind of assuming that there exists some distance function that knows how to compute this, and we'll talk at the beginning about how new functions that are not part of the language or not part of relational algebra can be registered in the system, and that's this notion of user-defined functions, but trust me for right now that these things can exist. Okay, and in fact, they don't have to be user-defined. There's many functions that already available in databases for manipulating say, for example, strings, and in fact, even uh for geographic information, there actually are distance functions available in most commercial databases. Okay, so you'll see this structure. The takeaway here is that I want you to still think join, right? Just because you don't see a quality condition doesn't mean there's not a join going on; it's just the same it's the same kind of joints everything else. And then the other thing, the other takeaway is just to know the term Theta join in case that comes up. Usually whenever when anybody's talking about the Theta join, what they mean is, you know, difficult joins, right? Arbitrary joins, gen the general case of joins. All right, so another example that's uh maybe you might sort of be able to think about coming up in practice in your own work is, well, you know, find all the user clicks made within five seconds of some page load. And this is sort of much like the distance argument you hold before, but now we can think about in terms of time, which is just a one-dimensional the one-dimensional district metric is easier to Define. So we say find the click time minus the load time of the page, right, so C dot click and P dot load and take that absolute value and see where that's less than five. And so this might be when you're trying to find people who find what they're looking for quickly, right? This is a this is a metric that that web analytics people might use frequently, right? If people sort of stare at a page for a long time, maybe if it's an article that might be good, and these are reading the article. If it's a navigation page, it may be bad; it means they don't and they don't find what they're looking for quickly. Okay. You might also hear about band joins or range joins, and this is things like find me um there might be an interval of Time start the start time and end time in one table, and you're trying to find tubals from another table that fall within that interval. Okay, and we'll actually see an example of that. There's other joints. Another join to to recognize that exists is this notion of an outer join. And here what you're saying is you want all the tuples from the left R1 R2. We'll write it like this with this sort of missing leg here. You'll want all the tuples from the left side if you've written it this way, and if the Tuple on the right-hand side matches, great, you put it out, and it's just like a regular join, but if it if there is no match, you still include the R1 uh Tuple, and you pad out the other the other columns with null as needed. Okay, so any value you don't have, make it a null. All right, and so the variants here that aren't particularly important is left outer join, right outer join, you know, so ah geez, left outer join, right outer join, and sort of full ladder join, which is a little bit hard to write because it looks like a cross product. The right out of join you really sort of never need because you can always just reorder the the operations. Pull out of Joy means that you want everything from both tuples padded out with null, and these are sort of ugly to reason about formally, but they come up pretty often in practice, because especially for users when they're well, I'd say especially for users as opposed to Applications, you know, if you're writing a query by hand, basically many times people find it surprising that that uh Records in their table disappear because they joined it with another table, right? But that can happen because you only you you've said that you only want pairs of tuples where some condition matches, and so you might you might have no matches and things disappear. And so how to join comes up as a useful way to match more with the what the uh SQL programmers expecting, especially with novices. Okay. All right, and so an example of this, we have two tables here, Anonymous patients and an anonymous job. We could do an outer join. Now, what what what what columns did we join on here? Well, it doesn't specify. We sort of omitted it here, although technically we should write that you know right there in the subscript of the join operator, but we didn't, but you of course you can probably figure it out. Well, so you look at the columns that they have in common, right? This has an age column, and this has a zip column, and this has an age column, and this has a zip column, and so it's actually on both of those columns, right? For every Tuple in P, find me a corresponding Tuple in an anonymous job J. Whoops, there's an extra in there where age equals 54 and zip equals 98125. So if we had just done a join, then this Tuple would be removed from the output because it has no corresponding Tuple on this side. There is no 33 98120, but because we did an outer join uh we do include the Tuple, and we patted it with a null here. Okay, and we write this in SQL. I wish I'd included this. We read this in SQL. Lean back up a couple of steps here, and I'll show you. So just like we have um join here, you can actually write outer join explicitly powder if you wanted to. I'm not going to leave that in the slide because it'll be confusing. This is out of context, but uh and in fact, you can say left outer joint. I guess anyway, to match our example, put in the slides. The database that you'll be using in the assignment is SQL Lite, which has some nice properties for a single user case. The entire database is stored as a single file, and you can pass it around and so on, so it's a good tool to sort of have in your toolbox, which is one of the reasons why I selected it for the assignments, but it actually has some limitations, and one of which is you can't express certain kinds of outer joins, a particular full ladder join.

Okay, so now I want to talk about how to interpret or give some examples of how to interpret SQL statements sort of in terms of relational algebra. We're not going to actually write out the plans, but I want to give you some experience staring at what may seem sort of complicated and kind of teasing out what's actually going on here. And so for people that have spent a lot of time around databases and SQL, these may or may not seem particularly complicated, but if you're just starting out, they they probably do. So in this first example, what do we see here? Well, what you want to look for when you're sort of staring at something that may seem sort of hairy is, you know, look for the from claws here. And so in this case, it's a little funny, right? Because we see whoops, we see that the from Clause does not have a table name mentioned; it has a nested query within it, and we remember that that's

Perfectly fine, because of this closure property of the underlying algebra, we know that any relational algebra expression and therefore any SQL statement is going to return a table, and it operates on tables. So if you can operate on a table, if you're operating on tables and you know you return a table, then you can sort of chain these operations together, and you have this nice closure property. So we know that we're allowed to query derived results just like we're allowed to query base tables. Okay.

And so in this case, we're doing a derived result now. While so let's go down to the other FROM Clause. Well, here we see another nested query, another layer of nesting, again perfectly fine.

And one more layer down we see this table here. And so where this data came from was a sensor that was mounted underneath an oceanographic research vessel that was collecting measurements of a variety of different variables. Here a few of them are mentioned: fluorescence, oxygen, nitrates, and there's and there's several more. And then this is latitude and longitude where the where the ship actually was located at that point in time, and they're also tagged with this timestamp. So they're tagged with latitude, longitude, timestamp, and a bunch of measured variables.

And so what this operation is actually doing is aggregating, binning these measurements onto five-minute windows. Okay. And so we can see how that's done. So in this inner query, there's some work to or they call this function to cast the timestamp to a float; that's not particularly important. And here we see this trick where we just use a constant value right there in the SELECT clause. And so what that does is uh sort of pinned a new column with a with a value five in every in for every record. Okay. I'll say maybe why this was done in this particular case uh in a moment.

And then in the next layer up, we see this kind of hairy expression involving bin size twice, and what it's doing is rounding down the timestamp to the nearest five-minute window. Okay. So you know, six minutes and 32 seconds becomes five minutes, and 11 minutes and 29 seconds becomes 10 minutes, and so on. Okay. Uh, and then in both cases, by the way, we see the star here, meaning that all other columns are going to be passed through.

Finally, the outer query, we have bin ID, which we computed here. Notice that we've got this renaming operator; we can have this complicated expression and just give it a nice convenient name. So you got that passed through, and then we compute the average of these other values, the average latitude and the average longitude within that five-minute window as well. Okay. So given that we're doing an average, we should expect to see a GROUP BY, and in fact we do; we're grouping on the bin ID, which would make sense, and then we happen to be sorting by bin ID just to make sure that the records come out in timestamp order because perhaps some application requires it that way. Okay.

So why is this why is this sort of overly complicated? Why not just collapse all this into one uh expression? Well, you could, but it's for the same reason that you might sort of abstract things or refactor things in an imperative language; there's a little bit of software engineering being applied here so that this complicated expression can be reused in multiple places. In this case, it's only being used once, so you could perhaps move it there, but it sort of separates two different blocks of logic. Okay.

So in this slide, I've I've color-coded the you know, three different blocks of logic: red, blue, and green, so you can see the layers of nesting. But the main thing I want I want you to take away is that one is that nesting is perfectly fine; you may see it and not to you know, worry when you see it being used in practice. And second, this trick is going to convey that as you start to do more and more analysis in SQL, more complicated analysis, you there are ways to kind of refactor the complicated queries so they don't necessarily look so complicated. Another thing you can do here that we will talk about in perhaps the next segment is save this result as a view, right, give it a name, and then you could refer to it in the outer query just as a table. Okay. I'll talk more about that next time, but these are some of the tricks you can play when you're sort of working with in in SQL, and in fact, you know, some of the things you may see people do even if you're not planning on doing as much SQL authoring yourself. Okay. Fine.

So here's another example. Same thing: the first step is to look at the FROM clause and see what you see, and here we see two tables, and there's this keyword INNER JOINs. Now the join is explicit, and the join condition here is this where we have some sort of ID and some sort of other ID. By the way, one of the other things I want to kind of do here is to show that you can kind of analyze the structure of a SQL statement to understand what's going on even if you really have no idea what the data is all about. And in fact, it's kind of helpful to do so. This is something you'll be presented with in a data science context as someone will say, "Look, you know, we need to know we need to predict what the average sales for next month is going to be," and you say, "Okay, great, you know, give me the data," and they'll say, "Well, I don't know; it's in some database over there," right, and she'll go over there and talk to the DBA, or maybe there won't even be a DBA, and you'll just sort of analyze what's going on inside that database on your own. And so that means staring at the schema, which we haven't done, but it also means staring at certain people at queries which you may not have written. Okay. So having a little bit of a skill of analyzing these complicated queries is it can be important. Okay. So fine. So this looks ostensibly like the join condition, but if you look at the WHERE clause, I want to make sort of a point here that the table referred to as X, this hotspot deserts, and the table referred to as W are involved in additional conditions down here. Okay. And so these are actually join conditions as well, right, even though this one was explicitly listed as all you know, INNER JOIN on this particular condition; anything that involves anything that applies a condition to attributes that are in both tables, right, any sort of condition of all the attributes from both tables is a join condition, and so this this whole thing is actually one sort of complicated join condition. All right.

So I went a little bit out of order into the a little bit different than the order I wanted to go in, but let me let me backtrack and come back to that. So hold that thought. Popping back up to the top, this other piece of complicated logic here, well, this is a particular syntax that's available in SQL called you know, a CASE statement, and it acts about the same way as a CASE statement in other languages, so that's not too bad. But in particular, even if you just ignore all this logic, you can just collapse all this down and say, "Well, look, there's some function that's computing Lin overlap," and where did I get that name? Well, that's what they name uh the result of this complicated expression, the length of the overlap. And in fact, you know, I happen to know a little bit about where this query came from; what they're working on is genetic sequences, and you may be able to deduce that if you just stare at this; there's snip region, which is stands for single nucleotide polymorphism, and there's strain, and they give you a hint, and BP is base pair, and non-coding regions are including positions uh gives you a bit of a hint if you if you've done any work in bioinformatics, but if you haven't, that's okay. Point is, Lin, length of overlap, appears to be the name of this thing, so it's almost like there exists a function linked overlap that involves these attributes, and we don't even care what's inside it; it's just a function, so that helps us sort of see the underlying Simplicity of this query in this case. Okay.

Then back to this other complicated expression, let me show you a little about what's going on here just because I think it's kind of a fun example. If you break these out into these three conditions and you happen to know something about where this data is coming from, you can see that what this is saying is, "Well, look, we want the start base pair from the X table to be greater than the double the start base pair from the W table, and we want the end base pair from the X table to be less than the end base pair from the W." So it's this picture, right, when the blue X interval, right, it's a sequence; the X table is filled with ranges, all right, starting in ranges, intervals, and the blue interval needs to be completely contained within the red interval, or the red interval needs to be completely contained within the blue interval, or the red interval needs to straddle the X-start base pair, which you can do if you stare at this at this condition. Okay. So they're doing kind of an interval logic right there in SQL. And so the point of maybe looking at this in enough detail to try to understand what's going on is a couple of things: one is this point about the join condition is that it's not you know, even if you don't understand what's going on, you can sort of see the structural details to understand that it's just a join condition, but also that you can actually do certain kinds of analytics directly in SQL. This is a fairly non-trivial operation to do that many people, especially among people who have either not had good experience with databases or have heard from their friends have not had a good experience of databases, is that this would be something that's considered sort of impossible, and it's not impossible, and nor is it really even a bad idea; it's actually kind of a natural thing to do. And so analytics in the database is should be a part of your bag of tricks; you know, the first step should not be, "Let's pull everything out of the database and then start using imperative code." Okay. This example alone, I would hope doesn't convince you of that claim I just made of of you know, getting things out of the database a good idea, but there's there there's going to be a sequence of arguments that I make uh probably throughout the quarter here here and there. Meanwhile, I should put the caveat: this is not you know, I'm not going to be pushing databases as the ultimate solution to data science, but by any stretch of the imagination, but I there is a role for it to play. All right. And so now with these you know, we collapsed this into one function; we can also collapse the join condition into these two things: it must match on on this uh CHR field, and then it must have this kind of overlaps condition be true that we saw in the last slide. And so this is just an example of a theta join where there's some non-trivial function being applied on each pair of tuples. Okay. Let me stop there and pick up on user-defined functions next time. [Music]

Okay, let me add a short addendum about user-defined functions. So we've seen a couple of examples of them; in fact, I'll click back here briefly. We pretended that there existed a user-defined function called overlaps and a user-defined function called the length of the overlap, and I wanted to point out at least in this context or especially in this context that you can indeed define these these kinds of application-specific operations. So they're called UDFs, and you can as a user, you can write one of these things; you can register it in the database, and then you can call it from within your SQL statements, and you can assign it; you can grant permissions for other people to use it as well. Okay. So the database becomes kind of a repository of user-defined functions that can be called from anywhere. And so there's three types to be aware of: one is scalar functions, the other is aggregate, and the third is table functions, and you can tell what they are by how they're used within the SQL statement. So a scalar function will appear pretty much anywhere any expression can appear. So if you can add two attributes together or you can subtract them or do any kind of arithmetic, you can apply a function. So this can appear in the SELECT clause, or if you're in the WHERE clause, or appear in a join condition; it can appear sort of anywhere an attribute can appear. Okay. An aggregate function appears in the SELECT Clause only and is always associated with a GROUP BY. And so here I've kind of indicated perhaps a common aggregate function that users define when it's not available. If you have a table of identifiers and words and you want to concatenate all the words together to make one long string, that's not always something available built in, but it's a very natural thing to want to do, right? So I've got a I've got a set of small strings, and I want to create one long string. Well, concatenation is something that's very easy to do in most programming languages, and that's how you would express it in SQL; you would do a GROUP BY, right? You say uh this example is a bit abstract to maybe go through live by waving my hands a lot, but if you've got some grouping attributes which defines a set of related strings, maybe I want to concatenate them all together, and so you might define a user-defined function to do this concatenation built from a string primitive that's just a PIN to PIN depend, you know, if you have if you have a function that concatenates two strings, you can build a user-defined aggregate that concatenates many strings. Okay. Fine. And then table functions appear in the FROM clause, and they're arguably the most complicated. So the most the most common example you'll see here is some kind of a table function that will allow you to find a sequence of excuse me, a sequence of integers. So sequence from 5 to 10. If you want to represent all the integers from five to ten as a table, you know, one thing to do is actually physically create a table on disk and insert those integers into that table, but that's a little bit wasteful, right, because we sort of know what that sequence should be; we don't actually need to physically store it. Five to ten maybe not so bad, but if you're trying to store the integers from one to a million, it can get worse. And so a function that knows how to generate these on the fly can be useful, and so you'll see table functions be be used in this way. Now uh well I I think it's up here on the on the next slide actually. So support for these are pretty comprehensive; all databases have them with unfortunately a notable exception being SQLite itself, which is the one you're using for the assignments. So I encourage you to go out and look at other databases; in particular, PostgreSQL and Greenplum, which is a commercial database that is parallel and based on the PostgreSQL code base, has really really excellent support for user-defined functions. SQL Server and Oracle, IBM all also have great support for it, but PostgreSQL has a particularly clean interface and was really designed way back when with extensibility in mind, so it was one of the first uh it originally was a research project, and one of the main goals of the research project was to show that an extensible database was a good idea; things they could you could add your own functions, add your own types, it add your own features in various ways. Okay. And so in particular, support for manipulating time data, timestamps, is is quite nice there. Okay. So you can define these these these user-defined functions in a variety of languages, including pure SQL, and you may have to sort of scratch your head and think about why that would be a good idea, but the short answer is that it's the same reason why defining functions is a good idea in any programming language, right? It's this notion of abstraction. Okay. And reuse. Okay. And then there's programming language, imperative languages that are extensions of SQL, and Microsoft's got one called T-SQL, and Oracle's got one called PL/SQL, and PostgreSQL has one called PL/pgSQL that add things like we'll run a query and save the result in a variable and then reuse that variable later, or we can define variables of just primitive types like integers and reuse those, and there's usually also looping constructs and conditionals and every other kind of feature you might have in an imperative language. Okay. So these come up, and they're useful, and if you have if you're a database administrator or you have friends that are database administrators, they're going to be very familiar with these languages. I'm not typically a big fan of using them, not because I don't think they're a good idea, but because I find them to be overused; I think there's things you can do without having to drop down into writing imperative programs that are not always recognized. Uh, the other reason I don't like to use them is is not for any kind of fundamental reasons, just because the experience of writing code in these languages is actually kind of painful; they're difficult to debug because they're not there's not great support for debuggers uh when things go wrong uh uh at run time; they're sort of hard to hard to figure out what's what's happening, and and so on. And while while I complain a lot when I see SQL logic being pushed into the application layer, I complain less when the alternative is to do it in one of these programs, imperative extensions to SQL. Okay. Don't put your joins in the application, but you can put your loops. So fine. So in Microsoft SQL Server, any kind of CLR language from .NET can be used, and they're typically in C# and so on. Python; there's an R extension to PostgreSQL that I haven't had much of a chance to play with yet, but that's kind of exciting because it gets the folks that are interested in using databases with their statistics routines a much more compelling argument of how to do so. Okay. So that's what I want to say about that, and we'll see examples of user-defined functions come up in a few different places, and most importantly, they'll all use the term, and so I want you to know that the term comes from databases, and what I mean is things, just any kind of code that is not provided by the system itself. All right. [Music]

Okay. So we talked about algebraic optimization, and then we talked about declarative languages on top of the algebra in order to simplify expression and in order to avoid specifying to the computer exactly how to do it, right? We want to leave that open and let the database figure that out, but we stopped at what I'll call logical optimization, and when I want to talk a little bit about the physical level of optimization, and what I mean by this is that even after you specify—we hinted to this last time—but even after you specified the order of operations, we haven't yet specified every detail needed in order to actually evaluate the query. Okay. And let me give an example of that. So here's a simplified version of a query we looked at last time where we say for every order we want to find all the corresponding items that were part of that order, and that's it. Last time we had an extra condition. Oops, I'm actually pointing with the mouse, but you can't see the X; I'm on the wrong screen. So so for every order find the corresponding items that match, and the last time we had another predicate down here, and this time I've taken that out. And so the algebraic plan that this translates into is very simple; it's just a join of the two tables, and that's it. So you think we're done, right? We're going to join order and item, and we're finished. Well, we've got to specify how we're going to do that join. And so let me tell you about a couple of the options here. So one in sort of very high-level pseudocode looks like this: we could say for each record I in item and for each record O in order, check to see if those two records agree on the order fields, on the order attributes, and if so, return it, and that's that's a join result. Okay, you know, they match, so fine. Another option is for each record I in item, insert that record into some sort of data structure, and here I'm going to call it a hash table. Now I'm not too concerned about what exactly that that is, and then second, for each record O in

Order go look up the corresponding records in that data structure that we found or that we built and return all the matching pairs. Okay? And if it actually is a hash table that we're talking about, then this lookup could be pretty efficient, right? It could be uh constant time, amortized constant time, right?

And so now this one says, well, for every record and item, go scan every single record in order. And so we have kind of an N squared complexity going on here. And here we say, well, for every record and item, put into a data structure and then, after that, for every record in order, go look up those records in the hash table. And if indeed this is constant time amortized, then this is a sort of linear time algorithm. So there's two different ways. So I argue that there's two different ways to implement this join, that both of these are valid. Okay?

So which one is faster? Well, I've sort of hinted that perhaps option two is faster, but in practice it may or may not be. And so, you know, I would I would pause here and ask the class to answer the question, but since it's over a video, I can't do that. I'll give you a moment to think about that, but I want you to think about why one, why this one for in particular might be faster in some cases than uh this one, even though it seems like it should never be. Okay? And let's see an example maybe in a second.

So leaving that question hanging open, I want to make the point that you have access to this underlying algebra. This isn't something that's all that's purely sort of theoretical, right? This is this is something that you can use tomorrow if you work with databases that you're at your job, for example, in this particular product, Microsoft SQL Server, and in fact, in all the products you're going to use the same sort of mechanism, but you can explain a query and that will give you access to some form of this algebra that I've been talking about. Okay?

So if you take a query, and here I've changed the query, I've changed the uh schema yet again. This table, this Reuters table, is one you'll be working with in the homework. Uh I've written a query here and I've explained it, and what shows what what the SQL management Studio gives back to me is a little algebraic tree, kind of like the ones I've been drawing here, just you know, in PowerPoint. Okay? And so this one says a hash match is going to be used to implement this join condition. This one's kind of a complicated join condition for a reason I'm not going to explain right now, but it has two leaves and then they get joined with this thing called a hash match inner join. Okay? And so this is very much like the hash table example I gave on the previous slide, but I want you to take a look at something.

So here I've taken the exact same query, but I've added an extra condition where I'm only looking for words equal to Parliament. And I probably should explain this schema a little bit. So the Reuters data set gives you term frequencies. You have three columns: doc ID, or let's just say Doc, term, and frequency. How, and the frequency is how often that term appears in that document. Okay? So this is the the table you'll be looking at, right? And so here what I've said is I'm looking for pairs of terms that uh that that co-occur in a single document is the is the previous query I was looking at, and now I've said, well, look, I don't want all pairs of documents or all pairs of terms, I only want terms that co-occur with the term Parliament, right? So perhaps lawyer co-occurs with Parliament frequently, so I'm looking for all the all the terms that co-occur in some document with Parliament is what this query is expressing. Okay?

So now what I want you to notice though is that when I explain this query, I get a different physical plan. The logical plan looks the same; it's still got scan, scan, and a join, but the algorithm to compute the join has changed, and now it's this nested this thing called nested loops, and that nested Loops corresponds exactly to this pseudocode here, that's why they call it nested Loops, the outer loop and the inner loop. So exactly the same thing. And so it chose to do this this nested Loops plan, even though uh we argued that it was an N squared algorithm and it should it probably wouldn't be choosing or chose very often, chosen very often. So why was it in this case? So if you think about it, the one of the sides of this join is only dealing with those terms or with it with the occurrences of the term Parliament in a document, which is a very small relation. And so it's a very small relation, and this nested Loops algorithm could be very very efficient and faster than dealing with the overhead of actually constructing this hat this hash table or constructing some data structure.

So the main takeaway here, as opposed to the details, are is that different physical algorithms are appropriate at different times, and that this declarative language and thing thanks to the declarative languages and thanks to algebraic optimization, the programmer doesn't have to worry about any of that; they don't have to make that choice. Okay? So this is a very very powerful idea; you just expressed the query and the database does the rest. All right, so fine. And just to point out, this is not just something you need to SQL Server; you can generate these kind of algebraic plans in postgres by using explain, and in fact they look kind of nicer, and here's these hash joins again. This actually shows you, whoops, excuse me, this shows you where it's building the hash table as the step one and then probing it with the step two, and same thing here, and this is another operator that we didn't talk about where you are say you're going to count all the records that match some uh for count all the members of some group, I'll put it that way. And so the hash here is on group ID, and you can apply aggregate functions to the rest of it, but I shouldn't give such a high level view of that without talking about anymore, so let me skip that all together. Okay?

So fine. So the algebra really does exist; you can look at it directly just by using the keyword explain, and I advise you to do so. If you work with databases, you should be using explain all the time to try to understand what's going on. All right. Another point I'll make is just that this matters. This is not from directly from SQL, and in fact it's not from a commercial databases, from some research that we do in my group, but the point is the same here. These are actually different physical plans for the exact same query, and in fact here I'm doing something in parallel, so this is actually a number of processors being applied. And so as you go from 4 to 16 processors, things go down a little bit, not as much as we'd like, actually they should be sort of going down quite a bit, but the point is that each one of these plans is doing a very different amount of time. Well, these two, these are kind of the same, but the difference is pretty important. So ignoring these opportunities and sticking with only the plan that the programmer specifies would be a big mistake.

Okay? And then another illustration of this that's a little bit hard to stare at, but let me let me give it a whirl to try to explain what's going on here. This is by some very nice work by Haritza et al at the LDB 2010, but there's a whole series of papers on this work, but they try to visualize the space of possible query plans. And so what the two axes are here, this is all for a single query, but the parameters to that query are changing. And so this in fact says something about the supplier account balance, and this is a parameter on the sort of extended price, and they change the value of these parameters in the query. So so imagine the same syntax, the same select star from something something where some condition equals extended price and some other condition equals account balance, and just by varying those two knobs you get this really rich tapestry of different plans uh being selected by the optimizer. So each color in this space represents a different query plan, a different algebraic query plan selected by the optimizer. Okay? And so I think that the takeaway here is just that this it's a very complex decision being made by the databases, and necessarily so. The actually you know, these these different planes actually matter; they don't show that here, but you can actually show that this choice of plan tends to databases tend to do a pretty good job of finding the right plan, and that you know I argued in the last slide that this can actually matter, the difference in the in the time uh can be pretty significant. Okay? So leaving this kind of complexity up to the programmer is a can be a big source of loss, right? Hiding this complexity is a huge huge huge win. [Music]

Okay. Last time we talked about algebraic optimization, and I argue that all three of these Expressions, without going into a lot of detail, but I argue that all three of these were equivalent, and they differed only in the order in which things were evaluated. Here you evaluate this join first and this joins second, and in this expression you evaluate this join first and this join second, and here you sort of find all possible combinations of tuples and then filter that. And so if you don't understand exactly what's going on in these Expressions, that's okay; you're not going to know that you're able to talk about it invite this in this segment I think, but the idea, the takeaway here is that there's three equivalent expressions, and we don't know necessarily which ones uh which one is the fastest one to you to evaluate, but the database can figure this out and does every time you write a query. Okay? And that's this notion of algebraic optimization. Now we don't, you know, even if you're familiar with databases, you may or may not be familiar with the racial relational algebra, which should be strange because I've argued that it's you know the Hallmark of databases and totally fundamental. So why don't we think about programming databases in terms of writing relational algebra expressions?

Well, another good idea, another key idea that's associated with relational databases is this notion of declarative languages, and we what we mean by decorative languages is that you specify the answer that you want, but you do not specify anything about how to get it. And so a relational algebra expression actually does specify an order, right? As I showed on this this slide, here's three different Expressions that's indicating exactly which order to do every operation, that means that some you know who if you write if you write an expression like this, you're instructing the computer, look, do it in this particular order. Okay? And so these decorative languages say, look, we're just going to describe the properties that must be true of the result, and we're going to let the database figure out the right order in which to do this. And so here's a quick example. So imagine you have two tables: one is order with three columns: order, date, and account; and another table with item with two columns: order and part. And the semantics here is that this column indicates which order that item should be associated with. Okay?

And so if you want to say find all orders from today along with the items ordered, then you might write this query. And if you've seen SQL plenty of times before, bear with me, and if you haven't, then pay attention. So select star, give me all possible columns from the table order and all possible columns from the table item, but I only want records such that this condition is true where the order column from the order table matches the order column from the item table, right? And then further I only want orders from today, or order.date equals today. So this is just conditions expressed over the results without any kind of idea of what of uh how to actually get this answer. So what automatically happens is that this query is translated into a relational algebra expression along the lines of what we've already seen. You know, here I've sort of just done a cartoon where you can say scan the item table, scan the order table, select the record such that date equals today, and then perform the join, find all the records in order that correspond to the that have quote, you know, for each record in order to find the corresponding records and item that match on on on order. Okay? So this is happening every time you run a query again. So the SQL is the what, not the how.

Given another example, there's three columns: product, purchase, and customer, and this the uh underlining here we haven't talked about, but this is the indicating what makes the table well what makes these records unique. And so here the PID makes the product unique, the CID makes the customer unique, and the combination of PID and CID makes the purchase unique. Okay? So here's another SQL query: we say select distinct product name—why do I know it's the product name because I see an X here and I see an X here—and the customer name—and I know it's the customer name because I see a Z here and I see a Z here—this is an alias for the for the relation customer from these two to three tables where the product ID in the product table matches the product ID in the purchase table and the customer ID in the purchase table matches the customer ID in the uh customer table—and that's a typo; it looks like that should be Z, so maybe change that on your own slides—yeah, let me see if I can fix it now: Z.CID—and then we want we but now we want only the products for which the price is greater than 100, and we only want the uh customers whose city is Seattle. All right? So what does this say in English? Well, find me combinations of products and customers—it's a unique print combinations of products and customers—where the customers in Seattle and they pay for a product worth more than a hundred. Okay? So it's clear what we want, but it's unclear how to get it; it's kind of a complicated query. Okay?

So translating this into relational algebra, we have this. So at the bottom we have product and purchase, and now we do this join where we say for every product find me the corresponding records and purchase, then we do another join for every record in the inner in the result of this join find me the corresponding records and customer, right? Now filter out all those records such that where price is not greater than 100, we only want the ones we're probably greater than 100, and we only want the ones where City equals Seattle, and then we want to in this case project um down onto these two columns—what I mean by project is get rid of all the other columns because except for the two we're interested in—okay, and then finally take the take the final answer. So the two points here is that the execution order is now clearly specified, but there are a lot of physical details are still left open. You know, this is a very high level indication of what's going on; order of operations is cleared, but that's about it; we don't know how we're going to do the join exactly, and there's multiple ways you can do it. I've indicated that you know we're going to take for every every record in product we're going to look up a corresponding record and purchase, but we haven't said precisely what that means. Okay? I'll give it I'll give an example of this in a second.

So another example here, we only have a single relation called R, and it's got three columns: subject, predicate, and object. And you see this kind of schema when you hear about uh when you work with RDF data, the resource description framework. And RDF is a language and formalism and software stack for managing what is called linked data, and here sort of everything is it's it's a set of all facts; you can any kind of fact you can come up with you can encode an RDF; you can say you know the instructor of this course is Bill Howe, right? So here the subject might be this course, the predicate is has instructor, and the object is Bill Howe. Okay? And so this is a people use this this formalism as a very general way of encoding any information from any from any source, and we will make we might touch on this much later in the course. Okay?

So here's kind of a complicated query, but what it says is I'm going to have three instances of the same relation, and I'm going to join each I'm going to join them all up, and I'm going to look for a sequence of tuples such that we have a a person who knows another person who holds the account of a company who has an account homepage of a particular value, right? So you're looking for this sequence of where this Edge is knows and this Edge is holds account and this Edge is account home page. All right? Find the all possible combinations in this table where I've got you know finally all a b all instantiations of a b c and d such that this pattern matches and the joins are specified by these conditions: the object of the first relation must be this; it must be equal to the subject of the second relation, and the object of the second relation must be the subject of the third relation. Okay? So I'm looking for patterns in the graph that look like this, and in relational algebra you see this you this this query gets translated into this form: there's a selection to find predicate equals knows, there's a selection to find predicate equals holds account, and there's a selection to find predicated equals how count home page, and then you join and then a sequence of joins, and finally a projection just to pull out the the final answer that we're interested in, which is a slight clause. Okay?

So perhaps it kind of a complicated example, but I think the takeaways here: I want to mention RDF because it might come by a company come across it again, and I also want to demonstrate that you can access the same relation more than one time in a single query, and then I wanted to give another example of translating even complicated queries into relational algebra expressions. Okay? [Music]

Okay, so we talked about physical data Independence and we talked about algebraic optimization. I want to talk about another kind of data Independence, which is logical data Independence. And so, you know, we argue that physical data Independence was this ability to insulate applications and protect applications from changes in the physical organization of the data, right? So things were rearranged on disk; we want we don't want to have to rewrite all the code in the application, and this is what databases provide, and relational database in particular do a great job of providing this. But if you go back to Ted Codd's first paper and the quote even the quote I gave you, he talks about you know insulating applications from the internal changes in representation represent changes to the internal representation, but also insulating applications from changes to some forms of external representation. And what he means by external is things like adding a column to a table. So this isn't an internal shuffling of the bits on the disk; it's actually a logical change to the table; there's more data there than there was before, but you know, if you think about it, if your code doesn't care about that new column, you shouldn't have to rewrite it just because there is a new column. Okay?

So the ability to provide this logical data dependence is provided by this concept of views, and all relational databases have this concept, and somewhat surprisingly I find them to be somewhat underused in practice, right? And while uh okay, so if you're using them right now, if you know what they are, great; if you if you're using them even better if you use databases, but I've never heard of views, then this is this is a great time to to learn about them. Okay? So what is a view? A view is just a query with a name. So I write a query, I give it a name, and I put it in the database. Now I can then access that view as if it was a table in the underlying database itself, as if it was a physical table. So why can we do this? Well, I talked about this notion of algebraic closure before, right? We and and here and this is exactly what empowers what what allows us to do this. So we know that every query returns a table, right? We take tables on the input, we do some manipulation of them, and we produce tables. So we say that the language is algebraically closed, and so any any

The result of a view will always be something that we can then add other queries on. So we can stack queries on top of queries on top of queries on top of queries. Okay, so why might we want to do this? So one reason is to protect the underlying data. You can assign permissions to tables. So, for example, if you only want a particular user to see data associated with their account, you can write a view that filters everything out except for their account and then grant them access to that view. The most direct benefit of this is that it allows you to expose data according to a logical organization that makes sense for the user. So even things as simple as hiding some join—if you decide to reorganize your data into two tables requiring that programmers use joins to link them back up again—you can simply write a view that hides that join and let everyone access the result of it.

Now maybe this—this may sound expensive, but the cool trick here is that because of this algebraic closure, what happens is the user's query gets composed with your query that defines the view, and the whole thing gets sent as one big block to the database for evaluation. So the database simply doesn't care whether it came as a view and then your user query or whether it came all as one—always one query directly from the programmer—it's going to optimize it the exact same way. Okay, so this is—there's nothing but a benefit here. So let's see an example.

So given the schema purchase and product, define a view called store price with two columns, store and price, that has this definition: so select store and select price from purchase and product where the product IDs are equal. This is a little funny because we didn't put Pete—we didn't put pids here. So this is a little bit wrong. So assume that each one of these has a—that's assume that this is PID and assume that this is also good, and then it matches the query down here. Okay, so this result is now like a new table, and just like I said a second ago, you've now hidden the join from the users. And so complexities like these column names, perhaps you can insulate your users from, and you can name them whatever you want. And so this allows you to put this—this is what this logical data independence means—is that even no matter how I want to logically organize my tables, I can—I can expose a different perspective on the data than I—than I want to have myself. And so this separates the people who are administering the data from the ones who are actually accessing it. Logical data independence—key idea. All right, all right.

So how do we use a view? Well, as I said, all you have to do is reference the view in a query just like it's a table. And so here, if we want to define the notion of a high-end store and we say, well, that's any store that has sold some product over a thousand dollars, and you know, for each customer we may want to find all the high-end stores that they visited, well, being able to—being able to directly reference the store price relation there—do we—we defined in the previous slide—this view helps simplify this query, right? And so you can just write a query that directly accesses that view as if it was a table. And okay, so how is this actually evaluated? Well, that's actually—whoops—that's actually what's really fantastic about databases is that this query will just be folded together with the view definition and pass to the database where the whole thing is optimized in one go. Optimized in one go. So you don't need to worry about the difference between having a stack of five views and all being compiled again—are all being folded together in one query—the database doesn't care; it's going to translate the whole thing into one big query—exchange that for an algebraic expression—and then do the normal optimization procedure to come up with the best possible plan. So it's basically like free abstraction, right? It simplifies things for the—for the programmer without any kind of performance cost. Okay.

Now you can actually get better performance than writing the whole thing by hand. You know, it's equivalent to writing the whole thing by hand; you don't pay a penalty, but you can actually do better than that with views in some cases by materializing views, and we're not going to talk too much about that because they're sort of—is very specific to databases, and we don't see it quite as often in this broader context of data science that we're trying to talk about, but it's a good trick. And once you have the mechanism to store views, you can essentially cache the results, and that's what we call materialization.

Okay, so the last key idea I want to convey about database is that of indexes. So while indexes are certainly not unique to databases, databases are perhaps unique as a platform that can make them very easy to apply and deploy and automatically take advantage of. And so databases are especially—but not exclusively—effective at sort of needle-in-the-haystack problems—looking up individual records or small amounts of records from large data sets. They do other things very well too, but this is one thing that makes that they're quite good at. And the reason is that they can apply—that you can apply indexes. This second was a little bit of context, but what I mean here is that if you're trying to write code to do this yourself in say some programming language like Python or C or R, you're going to be a slave to what sizes of data fit into main memory, as we said before. Now you can absolutely be clever and start bringing in one chunk of data at a time into memory, processing it, putting it out to disk, and bringing in the next set and so on, but the code will very quickly become very, very complex. This is something the databases already know how to do, and so your query will always finish regardless of database size as long as it fits on disk, right? It doesn't matter how much memory you have available; it'll eventually finish. It may not be that fast, but it'll finish. They already know how to take advantage of main memory in this optimal way, and it's—and you know, it's not easy, right? It's a pain in the butt to try to code that yourself.

Okay, so effective use of—of the memory hierarchy, effective use of indexes—these are things the databases can do well. It's a great platform for applying these—these tricks. And so finally, you know, this—what I mean here is that the indexes are easily built and automatically used by the—by the optimizer. So to create an index, you ready—you can write a statement like this. Here I've changed the schema on you once again, but here we're sort of filtering on genetic sequences, and if I create this index, then this query will—you know, this—this very simple query is looking for all sequences that match a particular value—it will automatically take advantage of that index if it's there. You don't have to tell it to do anything. You write the exact same query.

[Music]

Welcome back. So this time I want to talk about what this term scalable means. We made the point that working with really large data is an important aspect of data science, and we've mentioned the word scalability before; we haven't talked about really what that might mean. So a couple different ways to think about it I want to talk about here on this slide are—are here. So you know, operationally and in the past, one way to think about this was, look, it needs to work on data that doesn't fit in main memory on a single machine. Okay, so maybe you still may have one machine to work with, but this means that if it can—you know, you need to be able to bring data off of disk in pieces, operate on it, and then maybe write it out in pieces. Okay, and so a bundle of algorithms that could work on data in this fashion by bringing in data piece by piece to memory such that the memory footprint at any given point was small—this is something the database is provided, right? So you could write a query, and you knew for sure that it was going to finish as long as the data was there on disk and you had sort of a minimal amount of memory—at least—at least to get started. Okay, but yeah, and I might use the term out-of-core processing here. So out-of-core means it uses the disk to operate. So in-core means the entire—everything you're doing fits entirely in main memory. Out-of-core means you need to sort of work with the disk appropriately. And so databases were—the database community were specialists at out-of-core processing of large data sets, but increasingly this notion of scalability wasn't really enough. Okay, and so you saw this pretty acutely with websites that were coming online in the 2000s where, you know, one big server—no matter how—how big you bought that server—you couldn't bring data off of disk fast enough to meet all the requests. And so you had to start being sure that things were in memory, and the only way to do that is start adding more machines. Okay, and so increasingly—especially, you know, Google especially is sort of known for this—although many of the large media companies do this—is that scalable really kind of means being able to use up to thousands or maybe even more—or tens of thousands—of cheap computers and apply them all to the same problem. And so we might call this uh, scale out, while getting bigger and bigger and bigger main memory and more and more cores perhaps would be scale up. Okay, fine.

So another way of looking at this that maybe a little bit more precise is to think about this kind of in terms of algorithmic complexity that you may or may not be familiar with depending on how much computer science you've taken, but let me give you just a flavor of what's going on here. So in the past you might call an algorithm scalable if for—if given N data items your algorithm does no more than N to the M operations on it. Okay, so this may be in—maybe one—in which case it's a linear time algorithm, or it may be two in case it's a quadratic time algorithm, and so on, but this was deemed, you know, tractable, right? And so you'd prove properties about—you would prove that you could find an a polynomial time algorithm to solve some problem, and it was sort of thought to be scalable or—or assumed—you know, well assumed it was—that was the definition of what scalable was—things that were non-polynomial that took more than this—for example, exponential—where you might have M to the N—where exponential time algorithms, and they grew much, much faster, and they were—they didn't scale. Okay, but you know, this isn't a very tight bound on scalability in practice, right? A quadratic time algorithm maybe sort of feasible; you start getting into the fourth and so forth; it becomes pretty difficult to do for—for very large data sets.

Okay, so now you would say that it really can't just be N to the M; it's got to be N to the M over K—over some—for some pretty large K. So you have to have a lot of—K being the number of computers you can apply to the problem—and you had to come up with an algorithm that can really exploit this properly. Okay, and then one more point that I'm going to make now but we're not going to return to in this segment but—but I hope to at the end of the course is that, you know, it could be that soon even this isn't good enough, and for N data items you really should do no more than N log N operations. And so the N here means for every—for every data item that comes in over the wire. So this is—this is applicable to sort of streaming applications, and the data is coming in so fast that you only get one pass at it. So for every operation I have, I'm allowed to process that data item, and then I'm allowed to put it in some sort of a—of an index, and that's this log N factor. Okay, and so whenever you see log, you should think trees. So I'm allowed to take each item, inspect it, and work with it, and then stick it into some tree data structure, but that might be about it; it's just too big to make multiple passes at. Okay, and so examples of this might be this Large Synoptic Survey Telescope that we heard about—you know, they're taking sort of 30 terabytes a night; you can't sort of make too many passes on this data at one time. Okay, and so this whole area we think of as streaming data, which I guess I have written here, but I'll write it again, and we'll come back to some of the techniques—you're dealing with big data in a streaming—uh—context. Okay, so fine. So two different views of what scalable might mean, and we're going to talk in this segment to give—to give you some examples and some intuition for this—can we make use of lots of computers—uh—in this—over K. Okay.

All right, so here's a little example problem that's admittedly somewhat oversimplified. So we want to find all the matching DNA sequences where a set of sequence is a short string consisting of the letters G, A, T, and C, and you're given a short sequence, and you want to find all the ones that just exactly match that. Okay, so find me all the sequences that are exactly equal to this one that you're given. So how much do you do this? Well, with this little cartoon, imagine that each one of these black lines is a sequence, right? So this black line corresponds to this sequence, and this black line corresponds to the sequence, and all the other black lines are other sequences, and you know, think to yourself for a minute—propose an algorithm to—to find sequences matching your target sequence while making no assumptions about the data whatsoever. It's just given to you as a list. One thing you can do is do sort of a linear search, and so we're going to inspect the first item and we're going to compare it to our target sequence, and if they're equal, great, we found one, and if they're not equal—uh, what do we do? Well, we move on to the next one. So this is not equal, and this is a—this all happens at time equals zero, and then we move on to the next one, and so time equals one, we check another sequence and compare it for equality, and if it doesn't match, we keep moving on, and so and so on and so on until we get to time 17 where we find a match. And here I've said contains instead of equal; I guess I changed the meaning here, but so yes, we found a match, and we send it to the output. Okay, so how long does this take? How many operations do we do? Well, we did 40 records; I'm sorry, we were given 40 records in this cartoon, and we made 40 comparisons. So within records and in comparisons, we say that the algorithm and complexity is order N. Okay, so this is a linear time algorithm for this simple search and retrieval task. So the question is, can we do any better? And if you've had some experience thinking about data structures, taking some data structures classes, you should be thinking, yes, we can.

All right, so one way to do this is to sort the sequences. So how does this help? Well, certainly we can still do the linear time algorithm and inspect these guys one at a time, but we can also do something a little bit smarter. What if we start in the middle? So start in the middle and compare our target sequence to the sequence we found here, and they're not equal, but we can see that this one that we found is less than our target sequence. Okay, so we know that the sequence we're on is to the left of our target sequence; we know the target sequence is to the right—right? It must be somewhere in this direction. So we've just removed the need to check half of the data—right, 20 to 20 records. Okay, so now jump to the middle of this guy and compare again, and now we see that—well, boy, we overshot—this one is greater than this one. Well, once again, let's see here; we've removed half of these guys on the first step, and here we—now we've removed half of these guys, and now we know it's in this range—skipped it a half again—and compare this one, and now it's less than, so we're sort of bouncing back and forth around our target—let's see if I can draw this a little better than I did—cross those out, cross those out, and then I'll cross these out—and so we know it's somewhere on this side, and in the next step we find a match. Okay, and here if we have multiple copies of the same item, we know that they'll be—they'll appear next to each other, so we could just walk through the records gathering up all the ones that match if we needed to. Okay, so how long did this take? Well, here we still have 40 records, but we only made four comparisons, right? So within records, we made log N comparisons; we did—we—we navigated this sort of implicit binary tree; we did a binary search over this sorted data. Okay, and so this lookup was order log N. Now we did have to sort the data ahead of time, and if you have to include that, then that's an N log N operation, and we're not going to necessarily talk about, but once you have that sorted data available to you, it's now—uh—you know, only takes log N operations. And so this is perhaps far better scalability. All right, and this is a—this is a good trick, and it's such a good trick that it's been baked into many systems, especially—I'll argue—relational databases. And we made this point before, but I want to make it again is that databases are good at these kinds of needle-in-the-haystack problems, right? Extracting small results from big data sets; they can transparently provide this sort of old style of scalability—what I mean by old style scalability is that fits in main memory—as we've said a couple of times—your query will always finish regardless of the size of your main memory. In addition, you can—it makes an excellent sort of index platform—a platform for building and using and reusing indexes.

Okay, so relational databases are good at this old style scalability in the sense of—of out-of-core algorithms, and they're also good at this old style scalability in the sense of finding logs—finding logarithmic time algorithms. Okay, so indexes are easily built and automatically used when appropriate, and we've talked a little bit about this during relational databases. You can write a single statement—create index—uh, give it a name on a table and a column name, and it will sort records according to that column. Now the actual data on disk may or may not be physically sorted depending on the details of which system you're using in—how this is working, and typically in this statement it would not actually be moving the physical records around, but it would build an auxiliary index. But regardless, you get to take advantage of this logarithmic time access pattern. Okay, and so just by writing this one statement—in one line of code—you can create the index and take advantage of it. Okay, and then every query that comes afterward that needs to use that—that would benefit from using that index is able to—the optimizer automatically selects the correct index if it's appropriate to use. So this is much easier than you having to sort of rewrite your code by hand in order to either make it out of core, right, or to take advantage of an index. Okay, so when you're comparing relational databases to say scripts in R, scripts in Python, there's a lot of algorithmic work that's already been done for you that you're getting for free just by turning your problem into a SQL statement. Okay, fine. So you're—you're, you know, buying into some—buying into a lot of code if you can tie one arm behind your back and write it as a SQL statement. It's not just SQL versus—versus a much more expressive language like code; you're actually getting a lot of benefit out of doing that.

Okay, so let's look at another task called read trimming. So here we're giving the same set of DNA sequences, but instead of searching for one particular sequence, we're going to trim the final few base pairs from each sequence. Okay, so we're going to trim off a suffix and return the data set where each read is now just a prefix of a former read. Okay, so fine. So how do we do this? And the reason why you need to trim off the suffix—this actually comes up in

Practice and the reason is that the accuracy of the sequencer drops off fairly abruptly after a certain length of read, and trimming off the last several base pairs from every single read is kind of a standard pre-processing operation. Okay, fine. So how do we do this? Well, we can do the same trick that we have that we tried the first time with the search task, meaning that we can process each record in turn, one at a time, right? So at Time Zero, we can trim off the suffix here and just return TACCT. In time one, we can trim off this suffix and so on. In time 17, there's our old friend that begins with GTA and so on. But here, you know, unlike the search task, there's no index that's really going to help us, right? We have to touch every single record and manipulate it; right, we have to take a prefix from it and remove the subjects. And so the operation is fundamentally order N, right? There's not going to be an algorithm that is less than order N; right, you have to at least touch every single record. Okay. But can we do any better? Well, yeah, right? Processing the first task is completely independent from processing the last task, which is completely independent from processing this accusing task processing the record. Okay. So while there's no index, we can break this data set into pieces and process each piece independently. Okay. So imagine we take our single data set and break it into these chunks and assign each chunk to a different machine, or maybe a different processor to be a little bit more general. Now at Time Zero, we can process one sequence from each chunk all at the same time, and at time one we process the second sequence from each chunk all the same time, and so on. And so here, how much work did we do? Well, we do the same amount of work; we still process all 40 records, but how much time did it take? Well, it only took seven I say Cycles here, seven time units to be a little more general because we were given these six workers. And so the complexity here is N over K, right? For every item we can divide, we do on average marine items; we do in N over K time steps were completed the work. Okay. [Music]

Last time we talked about scalability, and we argued that scalability really means working in parallel. And we talked about this specific task, which is, you know, read trimming. Okay. So this is a bunch of small genetic sequences, and your task was to trim off the last few characters from each one. Okay. And we sort of showed that this is pretty simple to think about in parallel; you would divide the sense of the set of reads into chunks and put them all in a separate on separate computers and process them all in parallel. Here, you know, there's a function F that takes a single read and trims off the last few characters and returns the prefix, and you can apply this function in parallel, and you can get out with the data set you want, which is a set of trimmed reads. Okay. So let's see some some more examples. So a new task that was actually needed to be performed at the New York Times in 2008, and they have a couple blog posts about it that you can read, was in a simplified version was to convert a bunch of TIFF images into a different format. And what was really going on here is that they had uh digitized images from the newspaper along with some information about the optical character recognition, so some extracted text, and they wanted to turn this into a more web web-friendly format. And so they had to convert the images to a web-friendly format, and they also had to convert the extracted text into a little package of JavaScript code. Okay. They're getting get ready to put this stuff on the web. All right. So this is 405,000 images, which was quite a bit, especially at the time. Okay. But the schematic looks sort of similar, right? You take a big set of TIFF images and you split them into chunks and put them on a bunch of different computers, and you have a function F that converts a TIFF to a PNG and does the other work too, let's say. And what you get out of these data set you want, a bunch of PNG images, and they're distributed across these these machines. Fine. Let's look at another example, right? So now we want to run thousands of little simulations, and what we have are the parameters to each one of those thousands of simulations. And so an example of this at the URL here at the bottom of the slide is from a simulating muscle dynamics, and this comes up a lot; you have these Monte Carlo simulations that need to do; they understand sort of phenomenon stochastically by running lots and lots and lots of simulations with different kinds of different inputs and then kind of averaging the results. Okay. As opposed to modeling everything precisely, so that you want to run thousands of simulations. Well, you have a set of inputs, so parameters to these simulations, and you break them into chunks and put them all in separate machines and apply the function. And here the function is actually running the simulation, and what you get out is uh the output of the simulation distributed growth of the machines. Okay. So this, you know, a pattern should be emerging here, right? So another example. So imagine each one of these little bars is a document, and your task is just to find the most common word in every individual document. Okay. Same thing: distribute the documents across the K computers, and then your function F, now in this case, opens up a single document, figures out which word is the most common in that document, right, and just produces that word. And so now you have a big distributed list of pairs where you know the first part of the pair is the document ID and the second one is the word. Okay. So that could be useful, but it's a bit contrived. You know, consider a slightly more general program that computes the word frequency of every word still in a single document, right? So instead of just finding the most common one and producing that, now you're going to produce a histogram of the frequencies of every word in the document. Okay. So given this input, you produce all of these items, right, a set of items. The only reason I'm making this distinction from the last one uh is that the last one will, you know, took a single word; now we're taking this clear that that's allowed. Okay. Oops, the animation isn't here, but I don't think a lot of fixed in it. So you have millions of documents; you distribute them again. Now your function returns a set of word frequency pairs, right? But that's okay, and now we have, you know, lots of little lines here; each line if there's a single word, let's say, so they're not one to one anymore, but that's no problem; the function just returns a set of things. All right. So fine. So there should be a pattern here, right? We have a function that maps a read to a trim read; we have a function that maps a TIFF image to a PNG image; a function that maps a set of parameters to the simulation result, right, it's the simulation itself; we have a function that maps a document to its most common word; and we have a function that maps a document to the histogram of its word frequencies. Okay. So so good. So these kinds of tasks we think we know how to do in parallel. Given a big set of objects and a function that knows how to process a single object, you know, you should be able to think about how to parallelize this, right? But we can abstractly understand how this is done, right? Say that one more time: we have a big set of objects and maps single objects; the objects among the computers and the functions in parallel; this is trivial. All right. So what if we want to compute the word frequency across all documents, not just uh okay. So here, you know, if we have sort of these three documents, now we want to get a single histogram that counts out the number of times the word people appears across all three of them, the number of times the word government appears across all three of them, and so on. So let's go back to our schematic here. The pattern? Well, now we want to compute the word frequency across five million documents, and we can still distribute them among K computers, you know, so far so good. And then for each document we return a set of word frequency pairs, and now I've switched the notation here from F to map since we can consider this a map; that's the the the terminology I used. Okay. But now what do we do, right? So what we could get out here, what we will get out here is a set of frequencies, but that's not what we want; we want a one big histogram. And to build this one big histogram, we have to make sure that a single computer has access to every occurrence of some particular term. So if the word history appears in some document on this machine and it appears, you know, twice in this document and three times and two two times in documents on that machine and so on, we have to sort of group those all up and send them to a single place just so we can count them. Okay. So let's look at this again. So if we distribute the documents across these computers, we map apply our map function to each document in order to produce a set of word frequency pairs, and now we have a big distributed list of these sets of word frequencies, and now we want to get these workers involved in the process, and these guys are going to be the ones who count the occurrences of a particular word. Okay. And so imagine all these little colored red lines are occurrences a word such that all the red lines are represent a single word occurrences of a single word, and all the green lines represent a different word and so on. Well, so these guys are going to get sent to their respective location such that this worker is in charge of handling all the occurrences of the blue word, and this worker is in charge of handling all the occurrence of the red word and so on. Okay. So now instead of lines that go, you know, from one to one, we have lines that go from this one computer to a bunch of different computers. Okay. And so on, right? So you have to sort of shuffle the data to spray this data out across the network in order in order to regroup it. Now fine. So now that we have the data grouped the way we want and partitioned the right way, we can apply another function, which I'll call the reduce function, which in this case it does something very simple; it just counts them. And that allows us to produce our final result, which is oh, there are four green words and four red words and three blue words and so on. Okay. So now the schematic looks a little different; we have a sort of a two-step process. So we start with some large set of objects distributed over a bunch of machines, and then we want to apply some function F to each one of those objects, and that was our first step. But then the output of those functions are all going to be redistributed across the network and grouped to form groups. Okay. And then the second step is to process each one of those groups, so I'll write instead of F I'll write map here, and I'll write reduce here. Okay. And so that's exactly what MapReduce does, and we'll explain this in more detail next time. But the key idea here is that the user, the programmer, is going to write these two functions, a map function and a reduce function, which are serial. And what I mean by serial is they're not parallel; you don't have to worry about how to manage just how to program a distributed cluster within each one of these functions; you just write a function map that takes in an object of some kind and returns some other object. And I'm not using objects in the sort of object-oriented sense here; it's just sort of in the mathematical sense, just any sort of input to produce any any kind of output. Okay. And then the reduce function takes a set of objects, which I'll note this way, and returns some other kind of object. And actually this can be a set of objects as well, and in fact this can be a set of objects as well; maybe I'll switch colors here; this can actually be a set of things, and this could actually be a set of things. So for example, we saw a document returning a set of word frequency pairs, but you can think of it; it's honestly important to understand what's going on here. Okay. So this is this is this is an interesting hypothesis, right, that perhaps all distributed algorithms can be expressed as sequences of these two-step operations, right, a map followed by reduce, and then maybe more MapReduce, MapReduce, MapReduce as needed, and we'll talk about that in more detail next time. Foreign [Music]

We talked about parallel processing as a lead-up to MapReduce, and we ended up with this schematic here in the context of this example where we're counting words across a set of documents, which is sort of the canonical example to start thinking about programming and MapReduce. And so each one of these vertical black lines are presented a document, and we split them into smaller sets and send each one of those sets to a separate machine. And then we applied our map function to each one of those documents in turn. And so if the map function, if you recall, took a single document and produced a set of pairs, and each pair was a word along with a count of the number of occurrences of that word in that document; various variations on this that you can imagine. Now this word may have appeared in multiple documents, one here, one on this machine, one on this machine, and so on. And so now we need to group them all together onto a single machine so that we can count them, and that's exactly what this shuffle phase did. So here here I've written four different tasks processing, sort of looking like it's processing a single group at a time, but you know, no one should think about how many map tasks do we, how are we going to have, and how many reduce tasks are we going to have? Well, the map tasks are one per document; we have to we have to call the map function; how many invocations of the map functions are going to be? Well, we're going to call it once per document; how many invocations of the reduce function are going to be? Well, it's the number of groups that are produced by the output of the map function; in this case, it's ones per unique word appearing in any document. And so in some sense, the the number of machines we need to apply to this problem is maybe kind of predictable; in the map phase, it's it's corresponds to the size of the input data set, which we, you know, presume to know. But the number of reducers we're going to need is maybe not known ahead of time, right? It depends on the size of the map output. Here we might be able to reason about it because we maybe know how many words there are in the English language; we can assume that with a big enough set all of those words will be represented uh at least once. But in general, it's depending on the output of the map, so you don't really know. Okay. And so the only point I want to make is that we made a decision here to draw it as four different machines, but it may, you know, maybe it may be the same six machines you use the map phase and maybe a thousand machines and so on; you know, nothing's stopping you from sending all of the word occurrences to a single machine and having this one machine process the green group, then process the red group, then process the blue group and so on, or maybe you would do four at a time because there's four cores in the machine, but that wouldn't be as perhaps as efficient because it would be doing a lot of serial work. At the other extreme, you might think, well, we're going to need millions of tasks; let's allocate, you know, hundreds of thousands of machines to process them so that each machine is doing very little work. Okay. And that might make sense, but then the trade-off is perhaps sort of spinning up all these machines and preparing them to do the work. Okay. So there's a decision to make there, and we'll we'll come back to that. Okay. But let's talk a little more about MapReduce itself; this is I'm I'm belaboring this for a reason because what I want you to do is start thinking in terms of MapReduce; every problem you have, think what if the data set was absolutely enormous, way too big for one machine; how are we going to split it into pieces? And a very good way of thinking about how to split things into pieces is to think about how you'd write a MapReduce program to do whatever it is you're trying to do. And so this is yet again the same example, just drawn in a different way. So here the input is document ID followed by a value, and the value here is the entire text of the document, and the map function just to make this clear produces a set of things, not just one thing; then they're shuffled to produce this; so this is word one with a count of one, word two with a count of one, word three with a count of one, and so on. And then on the other side, what we get is word one with a group of all the occurrences, and then finally the reduce function counts them all up and finds that there's 25 occurrences. Okay. Okay. So I'm probably I guess if if this is completely obvious, you can always fast forward, I guess. One of the beauties of doing this online. Okay. So fine. So what is MapReduce? Well, that's the programming model we just described, and there's a paper in 2004 that's on the reading list that describes this, and there's a couple of key motivations for for doing this in that paper that I think sometimes get lost when you hear about the popularity of MapReduce today, and we'll talk about those two those two benefits in in a moment. So so one thing to realize is that MapReduce refers to the abstraction, and it's the name given to it by the authors of this 2004 paper. Hadoop is an implementation of MapReduce that came a few years later and was written by some people at Yahoo originally and then became an open source product that is amended by the Apache and has lots of contributors. Okay. So you know, the key idea for MapReduce was really this programming model, which it says here, right? Now it had a system with it as well, but the programming model, being able to express lots of different tasks and, you know, have some sort of implementation automatically turn that into a parallel job, turned out to be pretty powerful, right? This was an attractive way to to write parallel programs again because you didn't actually have to worry about the parallel; all you did write a serial map function and a serial reduce function, and the parallelism happened for free. And so the evidence that this is not so much about the system as it is the programming model is that you see MapReduce implementations appear in other contexts; there's people who have implemented MapReduce over GPUs; there's ability to MapReduce on multi-core machines in shared memory; there's people who implemented MapReduce on high-performance computing platforms on, you know, mo groups of mobile phones and so on. Okay. So this goes back to one of the motivations with this course, or I want to focus on abstractions where possible as opposed to tools. And so we're talking about MapReduce, the programming model, but we'll spend a little bit less time on the specific implementation uh Hadoop, although you will have a chance on optional assignment to work with a deep directly. Okay. So fine. So what is the data model of MapReduce? It's this bag of key-value pairs; I mean, my bag is a set that might have duplicates in it, right? And so we've seen that before; the document ID with the value that's a key-value pair; sometimes on the input will be a little sloppy and not worry about precisely what the key in the value is; for example, if you're just given a record, you can assume that it's say the the entire record is the key. Okay. Or a document; sometimes even we may not have an explicit document ID, but you can assume the URL or the file name or something is the key. The output of the mapper though, the distinction between King value gets really really important because that's what controls the shuffle as we've seen in that in that example. Okay. And so both the

Data model here is all about key-value pairs, and the input is going to be a set of key-value pairs, and the output is going to be a set of key-value pairs. The point is that the set of key-value pairs can get arbitrarily large; right, we're going to be able to process this set no matter how big they get. There is kind of an implicit assumption that the key and the value are small, and small here doesn't necessarily mean very, very small; it just means it needs to fit on one machine. There's no support for, for if value grows to be terabytes, it's it's not going to work. And so a document fits on one machine, that's okay. You know, a uh an image that's on the machine and so on. Okay, so fine.

So the map phase, as we've said, you provide a map function. The input is an input key and an input value, and the output is a bag of intermediate keys and values. It doesn't have to just produce, you know, taking a single input to produce a single output; it can produce a set of things, and we saw this with the word count example. A single document came in, but a set of things came out; that's okay.

And then the reduce phase, what you're given is an intermediate, the intermediate key—this will be the same intermediate key that was produced by the map phase, right, one instance of the same intermediate key produced by the map phase—and then a bag of values that were associated with that intermediate key, and they may have, and the key, the important thing here is that they may have come from any mapper whatsoever. The grouping into this bag of all of everything that shares the same intermediate key is handled automatically by the system. So the system will group all pairs with the same intermediate key and then pass that bag of values to the reduce function. The implementation details of whether this actually came into you as a bag or whether it's an iterator, if you're new, that if you're familiar with that term, that you could step over, is implementation dependent, but it's not important to think about; it's a collection of values. Fine.

So here it is all in one slide: the map function takes a in-key and in-value and produces a list, a bag of alkia intermediate value pairs; and the reduce takes an out key and a list of intermediate values and produces a list of out values. I don't think I like the slide; I think I prefer the earlier one. The one thing I will mention is that the terms map and reduce, I tried to motivate that in the last segment where you can think about uh converting a tiff image to a PNG image; you can think about a mapping, a function that maps every tiff image into some PNG image, and that's where the term came from. And if you look back, this kind of came from the functional programming community to use these terms; it doesn't precisely mean the same thing, but it's inspired by that. Okay, all right.

So here's maybe the implementation for the example we have, and a lot of times what I like to do is sort of ask you to pause and think and stare at this code for a little bit and think about what it does. Here we've kind of gone through the example a lot, so I'll I'll, you know, reveal the the the secret, but but it's still instructive to work through this for a moment yourself. And in fact, maybe I'll end this segment there, and you can stare at this and make sure that you understand what it does. [Music]

Okay, last time we talked about MapReduce and gave the abstraction and went through some examples, and we ended up on this slide, which is maybe the first time we've seen pseudocode or any kind of code that actually implements these map and reduce functions. And so I asked you to sort of take a look at this; actually, the end of the last segment, there are a couple of changes, mistakes that I fixed in this slide, so you can compare the two and see if you can figure out what the mistakes are and why I changed them.

Okay, so so let's walk through this. So what does this code do? Well, as I sort of gave away last time, this implements this word count application that we went through schematically, you know, with with cartoons, and you know, this is the pseudocode that actually implements that, are you know, example of pseudocode that could be, they could implement that. You can't execute this code since it is just pseudocode.

So what are we looking at here? Well, the input, as we said, the data model of MapReduce is key-value pairs, and so the input is going to be a big set of key-value pairs, and the map function is going to operate on one of these key-value pairs. And in this case, the key is the document name, and the value is the document contents, so it could be a big string; right, it's maybe comes from a PDF file or a text file or a web page or whatever. Okay, and so this code is pretty simple; it says, well, for each word w in the input value—so this sort of assumes that somehow you can iterate over all the words in the in the input value, in the text of the document—uh, without really specifying how, then emit a key-value pair where the key is this first element, which is the word, and the value is the number one. Okay.

Then the magic Shuffle phase takes over and groups all the key-value pairs that have that share the same key into a single group, and so all the occurrences of a particular word will show up as a group, and how that group is represented is as a key along with what we've called here an iterator over the intermediate values. And if you're not familiar with the term iterator, you can think of this as just a collection of values. Okay.

So as an example here, if you have the word, you know, the map function will produce pairs like this: every time it sees the word history in any document, it'll produce this, and then finally on the reduced side you'll have the word history here and a sequence of number ones. Okay.

And so what is this code do? Well, it initializes a final result to zero, and it says, for each value in this list of intermediate values, add that value to the result, and so here we just add them all up, and then finally we emit a final key-value pair, which is the intermediate key, the word itself, and the final result. And so maybe the output here is, you know, history 25. And we walk through this a couple different times, so I'm hoping this is pretty clear by now.

Now I claim that without changing this reduce function at all, you could make a change to this map function and get a significantly faster algorithm for computing this, so I want you to think for a second about how that might be done. So the thing to look at here is that, well, goodness, we're emitting a key-value, and start resting my hand on this, we're emitting a key-value pair once for every occurrence of a particular word, and each one of those key-value pairs has to be shuffled across the network and sent to the center of the reducer. So if we see the word history 25 times in a single document, we're going to emit 25 key-value pairs for that word, and they're all going to get grouped up by the shuffle phase, but we have access to the entire document here in the map, in this map function, so why not pre-count all the occurrences of those words and produce a different key-value pair, right, which means the word history appeared 25 times in this particular document I'm processing. And so now uh fashionist at 25—sorry, that's confusing; that's confusing; I didn't mean to make the same number as this—this is, you know, in our previous formulation this problem, it turned out that we we are we said that the word history appeared 25 times across all documents, and I shouldn't use the same number up here because that's that's that's pretty confusing. So let me change that. So here we say the word history appears five times in this particular document, and it appears other times in other documents. Okay.

So now we have only one key-value pair emerging from this document for the word history as opposed to five different ones, and overall across all the documents, across all the computers being applied to this problem, that's a significant savings. Okay, and then you know, double check to make sure that you don't have to change this code here, but you know, you you hopefully it's clear that you don't because you're adding the total value into the result, and so here instead of adding the number one 25 or sorry five times—well, sorry 25 times I guess—in the reduced side you're adding it uh some number fewer times, right? You're adding five plus ten plus three plus four and so on to get 25. Okay, so this loop, it is evaluated fewer times.

Okay, so the reason I want to go through that example is to demonstrate that you know there's two things: one is to try to think in terms of MapReduce and think about how you can cast a problem as operating on a bunch of chunks, emitting keys to to define groups, and then operating on those groups; but also that you know you actually have a lot of control over the performance of these algorithms by just modifying the map and reduce functions, right? So even though you aren't working on the system internals, you only have these two points of control, you can actually get very different algorithms, very different behavior, and different amounts of intermediate results being created and so on, just when these two two functions. And so you want to get a feel for not just how to express it in MapReduce naively, but also get a feel for how to do things reasonably efficiently. And in fact, this example sort of demonstrates one of the things you're going to be looking for is the bottleneck—often, not always—is the amount of data going across the network, and so if you can reduce the amount of output produced by the mappers, especially in terms of number of key-value pairs, you'll tend to improve performance—again, not always—and we'll see some more examples of this. Okay, so I'm gonna stop there, and in the next segment we'll go through a variation of this problem that chain that has similar characteristics but really just to drive home how to design use my MapReduce algorithms on slightly varying problems. [Music]

Okay, so let's talk about another example that is sort of similar to this very simple word kind of example, but it has a slight change. Okay, so now we want to maybe know the makeup of a corpus of documents, a set of documents, and try to understand the characteristics of the word length. And so now instead of a histogram on word usage, we're going to group things by the length of the word. Okay, so we want to know how many words have greater than 10 characters and so on. All right.

So you know, here we might group words into big words, medium, small words, and tiny words, where the big words are everything that's 10-plus letters, and the medium words are in red here, and they're everything from five to nine letters, and so on, and you can define your own sort of bucketing scheme or just not even try to bucket them and use the exact number. Okay, so you know, what what we're basically showing here—so I'm hoping that some of you are already kind of seeing how this is a essentially trivial variant of what we already just did—and this is some one of the points I want to make is that you'll see these patterns in designing MapReduce algorithms come up again and again. So once you try to get a feel for how to do it, it won't be that every new problem looks looks different, but the other the other things about maybe the next few slides I think is that it shows a little bit more detail in how these things are broken up. So for example, if this is a document, uh, what we hear, I guess the point I want to make is that before we sort of assume that every document was a small thing, but it's not impossible that you might have one document that's very, very, very large. This typically wouldn't happen with a document for various reasons, but imagine these weren't just documents but these were just big data sets of words, and so every document itself, each an individual document may be so large that it can't be processed by a single map function at a time, and so the question is are we are we stuck here? Is this is is the is the MapReduce framework broken and it's going to crash? And the answer is no. What will happen is when you load this large document into the system supporting mapper do so, we'll talk a little bit more about what that system, what this lower-level system is, what I mean by loading the file system underneath MapReduce, underneath implementations of MapReduce, when you do that loading the file, the data set, the file will automatically be split into chunks, and so we can pretend that this document was so large that it needed to be split into chunks. Okay, and so chunk one is this top part, and chunk two is this small part. Now if a small document is underneath the chunk size, then it won't get split, but a large document will. Okay, and so this could have happened with the word count example too; this is not something specific to word length obviously, but uh it's another twist that we're that we're exploring. So fine.

So now this top chunk gets assigned to map task one, and it produces this little histogram of accounts of yellow words, red words, blue words, and pink words, and we can imagine you should think about how the code might look if you were to if you had to write this, right? You would need to take the link, you know, iterate over all the words in the document just like we did before, but now instead of emitting a key being the word itself, you would admit you would count its length and put in a case statement and figure out what color it is in this in this notation and add that to account. So the output is these four key-value pairs, and in chunk two we do the same thing and produce a different set of four key-value pairs. Then in the shuffle step, all the yellow key-value pairs will be grouped together, and there's two of them, and all the red ones are grouped together and so on, and then in the reduce phase you'll add these two numbers together to produce 37. Okay.

So the structure here is really identical to word count; it's just basically a change to uh the map function, and in fact, in this case you could actually literally use the exact same reduce function, right? For every key, add up all the contributions of it; you would need to change reduce at all. And this is something else you'll see is that you know certain reduced, certain functions are more general than others, and you'll reuse them time and again. For example, counting things and adding things up is pretty common in these in these MapReduce algorithms, and so reduce, general reduce functions that add things and count things come up time and again. Okay, fine.

So word count, generally the canonical place to start when thinking about MapReduce; word length, very minor variation on that; so let's think of an another sort of minor variation on that, and this one in argument is even simpler than word count. So here we want to build an inverted index, and what an inverted index is, when you have a corpus of documents, you can presumably efficiently access any given document by its name, right? So if you want to look up a URL on the web, you you can do so, but if you but for a search engine, you know, very primitive search engine, you might want to look up documents that contain a particular word, and so do building this index to support word lookup to provide a list of document IDs that contain that word is called an inverted index, and it's one of the primitive steps in doing any kind of text retrieval kind of system.

Okay, so now you know, imagine we just had tweets instead of documents, uh, and the input here is that the keys are these tweet IDs that I've invented, and the the value is the text of the tweet itself. And so the desired output here is the word along with a collection of tweet IDs. Okay, so how do we do this? Well, the the, you know, again the code is actually simpler than it was before because now on the reduced phase instead of, well, so we'll think about it for a moment: the map phase, instead of instead of producing each word pancake in one for an occurrence, we won't do that; instead we'll put out pancake and the document ID itself, right? And all of these guys will be produced, and so this tweet one, the map, the map task that process this tweet one will put out pancake one, tweet one, love tweet one, and so on. Okay. Further, you know, an optimization here is if you see the word—I guess I should have had an example of this in here—but if you see the word pancake twice in the same tweet, do you need to put it out? Do you need to emit the key-value pair twice? Probably not, to build this index, because all you're trying to record is that the tweet contains the word pancake, not that it uh not how many times it appears. So fine.

So these key-value pairs get shuffled across the network, and now the reduced task, what does it do? Well, it's going to get a key pancakes—I should have an s on it I guess—and it's going to have an iterator over all the document IDs that contain pancakes, which in this case is tweet one and tweet two. So do we need to do any processing on this key and group of values? The answer in this case is no; the reduce function is complete is is nonexistent; there's nothing to do, right? When the group is exactly what you want in this case, and this pattern actually shows up somewhat often as well where you do want the map and you do want the shuffle phase to do the grouping, but you all you want to do is to perform the grouping; you didn't actually care about the reduce function, so you're not counting these things, you're not adding them up in any way, you're not doing any processing on the tweets; you're just you just want to emit that, and that's perfectly fine because this group of values is perfectly serviceable as a as a value itself. So if you are used to say relational databases, you know, nested structures, collections of values within a single cell and a table in a single row are generally just allowed, right? And this is actually first normal form if you're familiar with that, but here we don't care; it's any it's it's any key and any value; a key can have substructure, and a value you can have substructure. Okay, fine.

So that's how to build an inverted index in MapReduce; uh, let me stop there, and next time we'll talk about this relational join example, so this will be how to implement a join from a relational database as a MapReduce program. [Music]

So let's look at an example that doesn't involve processing a corpus of documents. So let's think about how to implement a relational, the join operation that we learned from the relational algebra in MapReduce. Okay, so here recall that you're given two relations, and a relation is a set of tuples, right? And you're trying to find every record in one relation that corresponds to a record in the other relation, right? And so here we're going to join on SSN equal to imp SSN, and actually it wasn't quite correct for me to leave this unspecified; there's a notion of a natural join that would match it up if the field names if they have to rename is matched, but here they don't, and so I need to be explicit about what I'm joining on. Okay. And so the output here is these three records. This record joins with these two, and this record joins with this one. Fine.

Now imagine that both of these relations are huge. Okay, so the map phase here is going to process every tuple in general, and we have a problem right off the bat that join is a binary operation, right? It has a two-input relations, the left relation and the right relation; you're trying to find corresponding tuples in one of the match that match the other, but MapReduce is a unary operation; it processes a single data set. So how can you, you know, at first approximation, to a first approximation, you can't express join in MapReduce, period. But that's okay; there's a there's a there's a bit of a trick here, and the trick is look, you know, it's

Okay, that imagine the data set here is just a big jumble of tuples. It doesn't matter what table they came from; we just lumped them all together into a single data set called tuples. Okay, and that'll be what we process with MapReduce. Okay.

And so here I'm asking the question, you know, what is this for? Well, this is a label that we've attached to every Tuple so that we can know where it came from, and we'll see how that's used later. Okay. Now I'm not being specific about how where you get this label or how you do it in MapReduce, but I want you to think logically that it's necessary, and in practice, it's not that difficult. Right, because you know, for example, if you're processing, you have distributed files from this directory representing all the employee chunks, and you've got distributed files in this directory representing the assigned Department's chunks, you can look at the file name to tell what table it is.

And so in the map function that you write, you can determine that aha, this has file name, you know, this has the employee relation name and it's in its file path, and I'm going to go and attach that as part of the record. Okay. And so we could write pseudocode for this, and in fact, you might as part of the an upcoming assignment. All right, so fine. So what's the map phrase look like of this relational join?

Well, for every record on the input, you're going to produce a key-value pair. We know we have to produce key-value pairs because that's how MapReduce works. So what's the key going to be? Well, I'm going to give you the key is going to be the join attribute, the attribute that you're joining on, and the value is going to be sort of everything else, right, the rest of the Tuple. And we can actually get away with removing this guy to save some space, but typically you wouldn't bother. Okay.

So fine. So given a tuple that looks like this with three fields, we produce a key of 77777 and so on, and the value is the entire Tuple and so on. And again, notice that both the tuples from both relations are all lumped into the same input. All right, maybe belaboring this, but I want to make sure that's clear. Okay.

So so far, we've done two tricks: one is lump everything together, and two is produce a key-value pair where the key is the join attribute. Now what happens in the magic Shuffle phase? Well, everything with the same key gets lumped together, as we talked about, and so now you get a reducer invocation that has, you know, one reduced certification for every unique key. So in this case, it'd be one for the 909 and so on, and one for 777 and so on, and the list of values associated with that key will be all the tuples that share the same join key. Now it doesn't matter what relation they came from; they'll all be in this list. So in this case, we get one Tuple from the employee relation, and we get one Tuple from the Department relation, and here we get one Tuple from the employee relation, and we get two tuples from the Department relation. Right.

And so now in the context of a single computer, we have everything we need to compute the join, and further, each one of these reduce invocations can be done on a separate machine. Okay, and that's how we scale. So now this reduced function that the program would have to write if the if you're implementing a join phase would have to take this key and take all these tuples and produce the joined Tuple, right, where these two attributes came from employee and these two attributes came from Department, and same thing here, employee Department.

Now do you want to think about what kind of operation you need to implement in this reduce function? Well, you've got us a set of tuples from one relation, and you need to associate it with every possible Tuple from the other relation, right, because we know they all join; they're all by definition they all join; they all have the same key; they all match. So if you have every member of a set paired with every possible member of another set, if you remember that's a cross-product operation. And so here again, we see relational algebra popping up sort of in a different context. Okay.

So I don't want to confuse you too much, but the overall operation we're trying to do is implement a parallel join. It happens to be that locally, right here inside one reduce function, we recognize aha, that's another relational algebra operator, a cross product, but there's more for abstraction purposes than for algorithm purposes; just want to point that out. All right. So again, you put on your relation algebra colored glasses, and the whole world, you know, these operators start to pop up everywhere. All right.

So fine. So let's do this one more time, just make sure it's clear. So I'm giving you two relations, order and line item, and they have these fields: an order ID, an account, and a date, and here there's an order ID, an item ID, and a quantity, and we're going to join on order ID. So what's the map phase look like? Well, once again, the key will be the join key, in this case, order ID, and the value will be this pair, the relation name along with the original Tuple. And before we just sort of lump these together in one Tuple, and it doesn't matter very much here, I've kind of structured it slightly differently, but the point is all the information is here. The key must be the join key, and the value is the Tuple as well as some sort of indicator for of what relation it came from, like maybe I'll ask real quick, why do I need that indicator of the relation?

Well, in the reduced phase, I wouldn't be able to produce the joint tuples properly if I didn't know which tuples went with which relations. If I just had a big bundle of tuples and I couldn't really figure it out very easily, that would be a problem. Theoretically, you might be able to avoid tagging it explicitly with employee and Department because you could deduce that well, the employee table is a one that has a string first followed by the joint, you know, followed by the join key, and the Department relation has the join key followed by some other string, but it's a little dangerous to rely on that kind of information because you could have you could be joining two tables that have exactly the same schema; there wouldn't be any real obvious way to figure it out, so an explicit tag is a little bit safer here. All right, fine.

So that's the map phase process, although we lump all the records together and produce key-value pairs where the key is the join key from the from the corresponding relation join attribute from the corresponding relation, and the value is the entire Tuple tagged with this with the relation name. Okay, and there it is, right there on the slide. Okay. So what's the reducer look like? Well, now we've got all the tuples that have to share the same key together, and it will produce these joint tuples; it will join this order Tuple with both of these line item tuples to produce these two join tuples. Okay, fine.

So now let's go on to a different example, and so here maybe we're going to get started with analyzing The Social Network. So here we have a graph where every Edge represents a let's say a friend relationship, or if you're thinking about Twitter, you can have this be a follows relationship, and so the input here is a set of edges with the semantics of Jim is friends with Sue or follows Sue, and Sue is friends with Jim or follows Jim, and so on. And so one point I'm making here is you'll notice that if Lynn follows if Jin if Lynn points to Joe, then Joe points to Lynn, and so this is there's a symmetric relationship here. Okay, and I've done that for Simplicity's sake to sort of avoid the confusion that can result from thinking about undirected graphs versus directed graphs. Okay, fine. So anytime you see an edge going one way, you'll see the other Edge coming back.

Now what we want to do here is well, before before I say what we're going to do is need to make sure the task is clear; we're going to count the friends. We want something very simple; we just want to say how many friends does Jim have and how many friends does Sue have and so on. And so the desired output here is Jim with three because Jim has two as a friend, and Jim has Kai as a friend, and Jim has Lin is a friend, and we want to have Lin equals two or Lynn has two friends because Joe and Jim and Chi is just one right here, Jim and Joe is just one, which is right here Lin. Okay, so it should already look like something we've already done before. So it happens to be social network analysis, and the records happen to be these pairs of people, but the algorithm you're going to use should start to stand out to you. Okay, and maybe you'll see why in a second if it's if it's not clear.

So in the map phase, how we're going to do this? Well, for every friend on the left-hand side, produce a key-value pair where the key is the name of the friend or the name of the person, and the value is just the number one, indicating that there's exactly one friend associated with that sorry, exactly there's that we think that we've encountered one friend associated with say Jim in this case. All right, so this record gets turned into Jim in one, and this record gets turned into Sue in one, and this record gets turned into linen one and so on. Okay, and then you know through the magic of MapReduce, the shuffle phase produces this intermediate result where we have Jim, the key associated with a list of occurrences one, one, and one, and Lynn is associated with two occurrences and so on. And then the reduce phase just simply adds up all these occurrences. And so what does this remind you of? Well, it's a lot like the word count example, right? Instead of documents producing words and counts, it's even simpler; it's for every record just produce a single count for that for for that person that the record represents, the person on the left-hand side. Okay, and after that, the reducer is literally exactly the same; it'll add up all the occurrences and produce a single number. Okay.

So in the next segment, we'll talk about something a little more, a little trickier, which is implementing matrix multiplication in MapReduce. All right. [Music] Okay. So let's look at a simple matrix multiplication algorithm in MapReduce. So before we get there, just to remind you how to think about matrix multiplication, we've got a matrix with four columns and two rows multiplied by a matrix with two columns and four rows, and the output is going to have the number of rows from the first Matrix and the number of columns from the second Matrix, so it's two by two in this case. Okay. And again, just to refresh your memory here, what is this result? Well, it's the first row of the first relation, you know, the dot product with the with the first column of the second Matrix, right? So it's 1.1 plus 3.4 plus 4 dot negative three plus negative two dot zero, and that should equal one. Okay, and so on. So that's let's just say Row one and column one dotted together gives you this position, Row two and column one gives you this position, and so on. All right. So I'm hoping that was intensely boring. All right.

So in MapReduce, how do we want to do this? Well, what we're provided here is two matrices representing sort of a sparse Matrix format, and a sparse Matrix format is going to look like this. Okay. So this is row ID, column ID, and the value, and the reason I call this a sparse Matrix format is that it would be inefficient to represent a very, very large Matrix this way if you had a value for every position, right? So if you think about just a multi-dimensional array in memory, you don't have to be explicit about the I and J coordinates; you only have to be you only have to provide the values, but if many of those values in that array are missing, then in this representation, I just can ignore them altogether; I just don't put them in, right? Okay. So anytime a value of 0, just remove that Tuple altogether. Okay.

So we're given two of these sparse matrices represented as tuples, you know, sets of tuples, and we're gonna do the same trick. Matrix multiply is a binary relation, and so we need to lump them all together, and we need to tag them with the source, and then we need to apply this kind of a trick in the map phrase. For every element i j of a emit several things: emit a tuple where the key equals I comma K, and I'll tell you what K is in a second, and the value is a sorry value equals a i J. Now we're going to emit one key-value pair of this form for every k in 1 to n, which is what I've written down here. Now what is n? Well, n is the number of columns in B, in the right-hand Matrix, right? So a is an L by m Matrix, and B is an M by n Matrix. So what does it saying? This is saying for every column of B emit a tuple with key I to K and value the value at i j. So I have a diagram on the next slide that explains this, but what you're doing is you're going to replicate this value to every column in in b. And then for B, you do the same kind of thing; you say the key is going to be equal to ik, and the value is equal to B j k, okay, and you're going to emit one of these key-value pairs for every I, the upside-down a, for for all for all I in 1 to l, where L is the number of rows in a, right? So you have to replicate the values of B to all the corresponding rows of a, and you have to replicate the values of a to all the corresponding Columns of B. Okay. So this and then finally, the reduce phase you simply you can you can do the dot product and produce the output. So it may be hard to think about written out a notation like that, so think about this sort of diagrammatically. The first the the I, you know, the the value I, the value 1 1 in a needs to be sent actually let me back up one step. First thing to recognize is that there's going to be one reducer per output cell in this algorithm, so here we're going to have six reducers, and if you have really, really large matrices, which is why we're playing this game is to imagine that we have, you know, ten thousand by ten thousand matrices, sorry, Matrix with Dimensions ten thousand by ten thousand, then this sort of made more sense. So one reducer per cell in the in the output Matrix. Okay. And think about what data does it need in order to compute its answer? Well, it needs for the reducer 1 1 in the output; it needs all the values from Row one in a, and it needs all the values from B1 sorry, from from the first column of B, right? All those need to be sent here. Now, fine. Let me Maybe I'll write that real quick: we need Row 1 from a, and we need column one from B. Right. Now let's think about this second position. Okay, so this is Row one column two. Well, here we need Row 1 from a in column two from B, right? So that's fine, but the problem is we don't have data represented in terms of rows and columns in the input; we have every individual cell, so we have to figure out where should this value a11 be sent? Well, it needs to be sent to everybody that might need it, which means it needs to be sent here because we see Row one from a, well, this is in Row one from a, therefore it needs to go here, and the second position is also involves Row one from a, so this guy needs to be sent to two places, which is what I was trying to draw here with these colors; it gets sent know let me draw some more arrows here and see because it gets too cluttered, which I'm sure it will, but we'll try it anyway. It needs to be sent to both of those locations. So for every column of B, you need to have Row this this value be sent, and similarly for, you know, this guy. Right. So to ever to all the reducers that might appear in Row that need the data for Row one, you need to send it to all of them, and so this is the reason I'm sort of belaboring this is that this is kind of a nice trick that MapReduce can do. Remember, you can you can you can replicate, right? You can send a single value out of the mapper to many places, and when I say send them many places, I don't mean literally sort of, you know, write it on the wire and send a packet across the network. What I mean is attach a value to multiple keys, and then every individual key will go to a different place into a different reducer. Okay. And through this trick, we can sort of arrange for Matrix multiply to occur, and this arguably scales pretty well, right? We've only we've done some replication, but that replication is kind of necessary, and this whole thing can sort of happen in in parallel. And then finally, again, each reducer produces the sum AI times BJ, sorry, it produces the dot product of a row of a and a column B. Okay. [Music]

So let's talk a little bit about what kind of systems MapReduce is deployed on. So I don't spend too much time on the system internals since this is a data science course, not a distributed systems course, but it's good to have an intuition for what's going on under the hood. So there's three types of systems to be aware of, architectures to be aware of: shared memory, shared disk, and shared nothing. In these diagrams, these cylinders or the disks, these rectangles are the memory, and these circles are the processors. Okay. So shared memory means that every processor has access to all of the memory and all of the disk, and this is what you think about when you have sort of a laptop. If it has four cores, and if you have a quad-core system, or nowadays if you have, you know, six or twelve cores in your laptop, this is the model that's being used, the architecture that's being used. Shared disk is somewhat less common in at least in these contexts that we're talking about, although certainly certainly common enough overall where you'll have individual machines that will access a shared file system. Now that setup is very, very common, but for using that setup for parallel Analytics is sort of the domain of high-end commercial databases, so you know your Oracles and your IBMs will often use a shared disk architecture. Okay. And then shared nothing is really what we're focusing on here, and this is what MapReduce is designed for and what increasingly and largely parallel databases are designed for as well. Okay. And shared nothing here means individual machines that are only connected by a network. Okay, no shared memory, no shared disk, but so the argument is that only the shared nothing architecture can scale to sort of thousands of computers and beyond. Okay, because eventually that the the sharing of memory or the sharing of disk eventually becomes the bottleneck and limits how many computers you can attach to the same device or same logical logical or physical device. Okay. And so learning how to program these massive shared nothing clusters is what MapReduce and parallel databases are all about, and the shared memory of machines are perhaps the easiest to program but are conventionally assumed to be pretty expensive. I should point out that you know the costs are dropping fairly quickly, so it's getting more and more feasible to buy a pretty beefy machine with lots of main memory and lots of cores, and you know your problem might fit inside that one. So you know when you see people deploying Hadoop and MapReduce on fairly small clusters on the order of say 10 nodes, you should see whether the data sizes they're processing are actually all that large, right? It could be that the data size of their processing something that fits in main memory on a similarly priced amount of hardware. Okay. So fine. So Hadoop and MapReduce are designed for really, really large clusters. All right. So this is the context we're in: a large number of commodity servers connected by a high-speed commodity Network, and here you know you think about a data center; there's a rack that has a number of

Servers and there's a data center that has mini racks, and this is how you organize your thousands or tens of thousands of computers. Okay, all right. So you're looking for massive-scale power parallelism that will run, you know, jobs that will run for many hours, even on thousands or tens of thousands of servers. That's really the context we're in.

So when you're in this context, an issue that comes up that does not come up all that often in a much smaller context is failures. Right? So if you're running a job for a long time on thousands of computers, the chance of something going wrong during that job becomes essentially, you know, 100 probability. Right? There's going to be something that fails, and so your system of processing of doing this sort of Analytics has to tolerate this kind of failure. You can't just—you can't roll back to the beginning and just restart every time there's a failure occurs, or you would never get anything done. Okay.

So, you know, even if the mean time between failure uh for say a disk is a year, if you've got 10,000 servers with multiple disk or ten thousand disks, let's say, spread across a thousand servers or any combination, you're—you're going to start to have failures, you know, once per hour. And you can look up the meantime's failure and actually do the math, but it's there's a—there's a couple of nice papers out there that I'll try to put in the readings if I remember. Okay. So failures are what we're concerned about here. All right.

So that's hardware. Popping up the stack one level is this distributed file system, and you might see HDFS too, which is the Hadoop Distributed File System. So remember the context here was that MapReduce was proposed in a paper in 2004 by Google, and Hadoop was the implementation of that—of the ideas in that paper. Okay. So we can almost use them interchangeably because the actual implementation at Google has certainly evolved since that paper and is not completely known. Right? So when people are talking about MapReduce, they're typically talking about Hadoop or other implementations of the programming model that have nothing to do with sort of scale out but shared nothing that produce can be assumed to be synonymous with Hadoop. Okay.

So this is a file system for very large files, and the idea here is that if you're going to take a single file on your own computer, you can manipulate it as a single unit, but if you're going to take a very, very large file and just—and you know, put it on a file system that's—that's on a cluster of machines, then there has to be some layer of logic that knows how to split that into pieces and put those pieces in different machines and keep track of where they are, and that's what this distributed file system software does. So each file is pretty, you know, as you're uploading this data to the cluster, each file is partitioned into chunks that are say typically 64 megabytes, although this is a configurable, and so each chunk—and this is critical—is replicated several times. Right? So there's not just one copy of the chunk; there might be multiple copies on different machines. Why? Because if one of them goes down, you want to have access to the other chunks. Okay, and you want to make sure these are spread across different racks in case the entire rack of computers goes dark, you still have another copy of it. Okay. And so the implementations here are GFS and HDFS. DFS is the concept, and the implementations are GFS and HDFS. All right.

So here's the phases of MapReduce that's a little bit more uh detailed than the abstract phases that we were talking about when we were talking about the programming model. Okay. So there's a file split here that's read from HDFS, and remember HDFS means is replicated; there's the—the chunks could be, you know, there's multiple copies of every chunk, and there's a unit of code called the record reader that breaks that chunk into—I'm using chunk and splits for this anonymously; I'm not a big fan of the term split because it's um sort of sounds too much like a verb to me—the record reader parses that splitter chunk into individual records. Then the—the programmers, you know, the user's map function is called on that individual record, and then there's a step called combine that we haven't talked about that I'll talk about in—in a moment. Next, actually. Okay. And then the output of these phases are written out to local storage on that node as we said. So then these regions in local storage, one per key, are pulled across the network by the reduce phase. Then all the regions from all the different—all the different map tasks that correspond to the same key are sorted together in parallel, and finally the user's reduce function can be called to produce whatever output it produces, and then the output of that step is actually written back out to HDFS so that it's replicated. So again, if something goes wrong in the map phase, you have to rerun the mapper, and if something goes wrong in the reduce phase, you have to rerun—you can pull—you can pull the output from the—from the local storage from the map phase uh, and if something goes wrong in the overall job, you know, you know you're safe because you don't lose data because the HDFS are—are—are replicated, and if you—you know, if you lose a reducer and you lose corresponding mappers, that's fine; you can sort of rerun whatever you need to rerun. Fine. So though the point is that you're—you're guaranteeing for fault tolerance during job execution. All right.

And so—so let's talk real briefly about the combiner. So to think about why we need a combiner, go back to this word count example that we began with. Well, in each case we produced a word and just the number one, and in the first—in the earliest version of this, we just produced the number one, right? Sending all of these occurrences of word one. So if word one appeared twice, then you're going to get a key value pair with W1 and a number one and another occurrence of W1 and number one, and you're going to send both of these guys across the network to be sorted in parallel and grouped in order to be processed by the reduce. Well, that's sort of wasteful; you'd like to combine these into a single record, W1 comma two, and then just send that because it's smaller. Well, you could rewrite your map function to—to do this, but it's such a common need that you can—that you have this capability called a combiner. And so a combiner identifies key value pairs of the same key and lumps them together before sending it over to the reduced side, so it just saves the network traffic. In many cases, the combiner function can be literally the same function as the reduce function; it all works out. And what needs to be true for this to work is that the function that you're applying needs to be associative and commutative, but I'm not going to say much—much more about that. Okay.

So here's what it looks like in pseudocode. We saw the map function earlier, and we were emitting this key value pair of a word in a count, and we saw the reduce function earlier, but we're adding new as a combiner function that has the same type signature as a reducer, and again, often—in many cases—can literally be the same—the same implementation as a reducer, and the only point of this is that it's being applied before sending data across the network. Fine.

So here's sort of a summary of a Hadoop job that I like a lot, and this is from a—Huevo—who is it now at NYU Poly? I believe he saw it in Hawaii, probably. So the data begins on HDFS, and there are these chunks, and the—in part partitions go to map tasks. Now, these are again not invocations of the map function; these are entire tasks, and the map—each individual map invocation may produce multiple key value pairs regardless; the map task almost certainly does. Right? So it's going to produce these local sort of colored key regions, and the regions are going to be sent across—pulled across the network to the reduce servers, and here in this example there's only two reduced servers. Now—now before we give this example, we sort of showed all the blue ones going together and all the red ones going together; that was a bit of a simplification. What actually is going to happen is that if you only have two servers, well, all the hundreds and hundreds of possible Keys need to be mapped to just those two servers, so you're definitely going to get a mix going to the same place. And this is where we've been a little bit glib up until now; we've—we've said that you specify the key and it hashes to a particular reducer, which is—which is true logically, but before that you have to actually get it to a machine where lots of reducing or lots of reduced tasks might be running or lots of reduced implications might be running. Okay. And so you know, in this case the blue—the blue guys and the red guys both ended up on the same machine, and the green guys and the orange guys both ended up on the same machine, and then this parallel sort manages that. Right? So it puts all the red things together and puts all the blue things together, and then for each individual color, one reduced invocation is called. Okay, and the reduce function is called, and it produces the output partition, and then that output is then written back out to HDFS. Okay. Let me stop there, and I'll pick up here and talk a little bit about parallel databases and how they do query processing with the point being that it looks a lot like MapReduce. [Music]

Okay. So let's talk a little bit about other large-scale data process systems besides MapReduce, and as a step two, that let's think about the design space of possibilities here. And so this is a breakdown that was proposed by Michael Azard, who developed a system called Dryad at Microsoft, which is a—a very nice system with the same sort of motivations as—as MapReduce. Okay. And so he divided this space into these three axes where you're worried about sort of low latency, very interactive sort of speeds, quick turnaround time versus things that maximize kind of throughput, you know, massive batch jobs operating on, you know, thousands and thousands of computers at once versus another axis here is sort of whether it's in a private data center or where they're scaled out widely across the internet. And then maybe the third access is data parallel versus the shared memory okay that we talked about. So the—the areas that we're mostly concerned with are going to be here, which is what we're talking about currently, and then in a couple of segments we're going to talk about these low-latency smaller operations, which you can think of as the NoSQL systems. Okay. And then maybe where—where Michael placed older relational databases, although he didn't label it the same way I have, is down here in this quadrant where they're mostly in shared memory sort of space and low latency. I—I say older databases to try to point out that not all relational databases operate in that space; many are—most, in fact, are data—most parallel databases are—are data parallel as well. And so this is, you know, MySQL and PostgreSQL, if you're familiar with those, are probably in this space, shared memory, shared disk. All right. And here HPC means high-performance computing, where, you know, it's private data center and it's a big massive shared memory, but there's—but—but it's a batch job submission system. Right? You say you run your big compute—compute-intensive job and submit it to um the machine, and it processes on it and eventually returns to the results. And then I'm not going to talk too much about this, but this notion of grid computing was really sort of focused on connecting up clusters and computers from different universities and letting them all sort of talk to each other, and so that's why he's pushed this up on this other access. Increasingly, you're seeing these systems also being pushed up in this access, internet-scale, planetary, you know, distributed hash tables with different kinds of layers for guaranteeing certain kinds of semantics. So the Spanner system from Google, fairly recently, is a nice example of this. Okay.

So sort of wrap up what we talked about last time: large-scale data processing, you know, many tasks uh need to process big data and produce big data, and so you want to use hundreds of thousands of CPUs or—and hundreds of thousands or tens of thousands of computers to solve these problems, but this needs to be much easier. And so there are such things as parallel databases. We talked about databases, and we sort of extolled the virtues of programming in that model uh, and they exist, but they're often expensive. Well, they—they almost exclusively are expensive, and they're difficult to set up, and it's actually not totally clear that many of the parallel databases scale to really hundreds or thousands of—of machines. Okay. And so MapReduce came around at a time as a bit of a response to this scenario. So it's more of a lightweight framework featuring, you know, automatic parallelization, distribution as we've been talking about, featuring fault tolerance, and I mentioned a couple other things here that are perhaps less important, but the I/O scheduling, the status of monitoring, to really sort of strip everything down to just parallel processing—not all the features that parallel databases offer—just parallel processing with the added benefit of fault tolerance. Okay. And this really seemed to scratch an itch with people. I'll—I've—I'll argue here, and I'll probably mention this again at some point, that it's not totally clear to me that MapReduce would have been quite so popular had there been available a parallel open-source relational database product, but all the open-source databases were all—not—not parallel; in fact, they were even single-threaded for query processing; you know, only a single thread was working on an individual query at a time uh, but that's—that's—that's speculation. Well, actually, I have a little bit of evidence for that, but I'll—I'll lay out for you in a bit. Okay.

So now I want to talk about—maybe I guess I'm building up to that argument—I want to talk about parallel databases and how they work and hopefully show that there's some similarities—so where there are similarities and where there are differences. Okay. So we're—called a key idea of relational databases was this notion of a relational algebra where you could write sort of plans like this, and that this top-level language called SQL was the most common way of producing a relational algebra plan. Right? You wrote a query in SQL, and it was automatically turned into a relational algebra plan by the system. Okay. So this is kind of thrown out the window with MapReduce arguably in favor of sort of flexibility in providing the programmer more—more control. But let's go back to this model for a bit. Fine.

So now we want to—we want to evaluate these queries; we want to do it in parallel now. And so there's two different terms that I want to be familiar with: one is distributed query and one is parallel query—or distributed query processing and parallel query processing. And so they're both ways of sort of taking advantage of more computing resources for the same query, but they behave a little differently. So in a distributed query, what you're doing is taking a single large table and distribute it across a cluster just like we talked about, and then you're breaking your query into individual pieces to operate on each of those uh partitions of the—of the file. Okay. So this sounds like—well, isn't that basically just the same thing as MapReduce? It is, except for the fact that all the results of those individual pieces are all sent back to the head node—to a single server—to sort of finish processing. So for example, if you're doing a count, right? You want to count all the records that match some criteria. Well, if you have a very large file that's split across several machines, these distributed query systems—Microsoft SQL Server in particular is—is an example of this—is smart enough to break the query into a bunch of little pieces and run each of those pieces in parallel, but as they start to stream tuples out to be counted, they'll send them all back to the head node. Actually, I guess that may not actually be true because it may be smart enough to figure out that you can just count things in parallel and add them up, but it's not hard to construct a query where you have this bottleneck of sending everything back to a single server. So it's essentially—you can think of it as like having the map phase but not really the reduce phase. Now, parallel query—every individual uh operator in the relational algebra is implemented in parallel. So when you're doing joins, you're doing joins across a bunch of nodes; when you're doing um groupings, you're doing groupings across a bunch of nodes, and we've seen how to implement a relation—relational joining in MapReduce, and if it's not too far off from how it's actually implemented inside databases. Okay. So if we know how to implement join—that's usually the harder one, trust me—that you can implement the other ones that way. Well, now we have a way to do parallel query processing with the relational algebra, you know, so why not do that? Well, the answer is people do—do that, and we'll come back to that in one second. Okay.

So for a distributed query—I guess I was waving my hands a second ago trying to explain this when it was on the next slide—you can imagine constructing a view, and we've talked about views; if you don't recall what that is, it's, you know, a named query that can then be accessed as a single table. So we say that the sales table is really the union of a bunch of smaller sales tables, one for each month, and in particular you could put each one of these sales tables on a different disk or even a different server altogether. Right? And then the—the user who's querying the sales table doesn't have to care about the fact that this is actually distributed—a distributed table; they don't have to go gather up all the results from January and then gather up all their juice holds in February and then gather up all the results from March and put them all together; that's done automatically by the system. However [Music] Right? And so this is—this is the CREATE TABLE statement that we didn't talk about for constructing the sales table for set—for individual March, you know. Okay. But again, however, when you process this stuff in parallel, that works great, but when you get the results, you need to send them all back to the single node to—for—to finish processing, and that's the limitation of distributed query. So it's great that you get some parallelism, but it can't do everything in parallel, and you can—you can see this when you run performance experiments. But a true parallel query example would be, for example, from a system called Teradata, which is a database company that many folks haven't heard of because they've been selling very, very high-end databases to very, very high-end customers, and so they don't sort of need to have much word of mouth in the popular news media. But what's happening here is that as every individual row is inserted into the parallel database, it will be assigned to some particular server using a hash function. Okay. Fine. So everything is automatically partitioned more or less randomly across the cluster, and then whenever you're running queries on this, all the—all the machines will access their—their data in parallel. Okay. And you can see this has a little bit of the flavor of how we did the relational join in MapReduce, and that should become even more clear in a second. Okay.

So remember we're—this is our query, orders and line items, and this is the plan that we're going to do: we're going to select some orders and then join the orders with the items. All right. So how this starts is these parallel processing units—units called AMPs—Teradata terms—will each contain a piece of the data, a chunk of the data, and the chunk was defined sort of randomly by hashing, and so they all—in parallel—begin to scan their—their individual chunk, and then they'll all—in parallel—apply the filtering condition to throw out certain records that they don't want, and then they'll all—in parallel—hash on the—actually, this isn't right; this should be hash on the order ID, not the item ID. So this is the join key. Right? We're going to join on order, and I probably got this wrong back here too. Yeah, this is wrong as well; this should be joined on order ID; it doesn't quite make sense to call this item. Okay. So then the all-in-parallel hash on the join—join attribute, the order ID, and that will shuffle it, for lack of a better term, to another set of AMPs, perhaps the same set.

Of amps, but typically another set of amps that will do the next step, actually compute the join. And so this should look a lot like a map reduce job, right? You've got a map function that's scanning and selecting, and and then and actually then hashing, and then we've got a reduce function coming up to actually produce the join. For the other relation, the same thing happens: you scan the items and then hash on again. This is order, order, order [Music] All right, so you're scanning, you're scanning and selecting on the orders and just scanning on the items, and then both are hashed on the appropriate join attribute. Lo and behold, uh, all the items and all the orders that correspond to the same order ID, to the same join attribute, same join key, end up on the same machine, and you can actually process the join just like the mapreduce example we saw. All right.

So then at the end of these two steps that I've shown you, amp 4 will have all the orders and all the line items where hash of order goodness equals one, and this amp 5 will have all the orders and light items where hash of order equals two, and this one will have all the orders and line items where hash of order equals three. And now it has enough information to individually and in parallel finish the join and actually produce the result. And all these ought to be orders as well. All right, so fine. So the point is that the same Machinery already exists in these parallel databases and in mapreduce. You know, if you're interested in doing a join, you're sort of implementing this uh yourself. And so this observation was not lost on people that, you know, hey, it might be nice if there was sort of a standard way of doing join in mapreduce and we didn't have to sort of rewrite it ourselves every time. And in fact, you know, there's there's libraries on top of mapreduce that do this. And so there's a library called Pig from Yahoo that I encourage you to check out that is recognizably relational algebra. Right, there are operators called join, there are operators called Group by. It does have a bit of a funny data model where you're allowed to have kind of complicated nesting as opposed to just straight tuples and straight relations, but the relational algebra is there. And in fact, this is sort of one of the points I want to make is that it, you know, it's important to sort of be able to modularize the concepts that come out of various communities, and especially databases. They it tends to be true that, you know, it's kind of All or Nothing. If you're interested in using databases, well, then you have to take everything; you have to take the whole package, you know, it's All or Nothing. But increasingly what you're finding is that these concepts are leaking out into other systems, which is why I'm really emphasizing this relation algebra piece a lot is that you can use these Concepts independently of buying in to a strict relational model, okay, and certainly not a strict adherence to a particular implementation of it.

Now another system called Hive is literally SQL on top of Hadoop, so it goes One Step even higher. Instead of just stopping at the relational algebra level, it actually provides a SQL interface. Impala is a more recent system from Cloudera. I should mention here Cloudera, by the way, is a company that has aligned themselves pretty closely with the Hadoop stack, and so they have their own Fork of the Hadoop system and a bunch of great tools for working with that ecosystem, and Impala is a new system that they produce that is provides SQL over hdfs and actually uses a lot of the code from The Hive system. Okay. Cascading is another system that's maybe a little bit less common, but it's also very recognized to be relational algebra. The Dryad system I mentioned has nothing to do with mapreduce directly except for a similar motivation, but it very obviously has relational algebra there. The Clistera system I I mentioned, it's a more of a research project, and it's not clear to me that the code is available, but it's also very clearly relational algebra. So you know, when you put your relational algebra goggles on, you start to see the world in this way, and it starts to come up everywhere. Okay, okay, so it's good to go back and understand those operations [Music] Okay, so I want to spend a little bit more time on the details of mapreduce versus already our versus relational databases, Beyond just the sort of how the query processing happens. We saw that parallel query processing is largely the same; some of the many of the algorithms are sort of shared between, and there's a ton of details here that I'm not going to have time to go over, but the takeaway is that that the basic strategy for for performing parallel processing is the same between them, but there's other features that relational databases have, and I've listed some of them here. So we mentioned declarative query languages, and we've mentioned that those start to show up in Pig and especially Hive. Now there's a notion of a schema in a relational database that we didn't talk too much about, but this is a, you know, structure on your data that is enforced at the time of data being presented to the system. So any data that does not conform to the schema can be rejected automatically by the database. This is a pretty good idea because it helps keep your data clean. It's not real feasible in many contexts because, you know, the data is fundamentally dirty, and so saying that you have to clean it up before you're allowed to process it just isn't going to fly, right? And so this is one of the reasons why mapreduce is attractive is it doesn't require that you enforce a schema before you're allowed to work with the data. However, you know, it doesn't mean that schemas are a bad idea when they're when they're available, and in fact, really, you know, even with mapreduce a schema is really there; it's just that it's hidden inside the application, right? So when you read a record, you're assuming that the first record the first element in the record is going to be an integer and the second record is going to be a date and the third record is going to be a string. So that schema is really present; it's just present in your code as opposed to push down into the system itself. And there's a lot of great empirical evidence over the years that suggests it's better to push it down into the data itself when and where possible. And in fact, you're starting to see this, so Hive and pig again have some notion of schema as does Drive link as does some some emerging systems. There's a system called Adapt that I'll won't talk about really at all, but uh, combined sort of Hadoop level query processing for parallelism and on the individual nodes there's a relational database operating, and one of the reasons among many is to have access to schema constraints. Um, fine. Logical data Independence. This actually you don't see quite so much, that this is the notion of views, right? Does the system support views or not? And you haven't seen quite as many instances of Hadoop-like systems that support views, but I predict they'll be coming. Indexing is another one. So we talked about how to make things scalable; that one way to do it is to derive these indexes to support sort of logarithmic time access to data that's not available in vanilla mapreduce. Every time you write a mapreduce job, you're going to touch every single record on the input; you're not going to be able to zoom right in to a particular record of Interest. That's wasteful, and it was recognized to be wasteful, and so one of the solutions you see people adding indexing features to Hadoop, and an hbase is an open source implementation of a another proposal by Google for a system called Bigtable that, among other things, provides kind of quick access to individual records. And hbase is designed to be sort of compatible with Hadoop, and so now you can design your system to get the best of both worlds. Okay, so you can get some indexing along with your mapreduce style programming interface. And once again, I'll mention Adapt here as well; one of the one of the motivations for Adapt to be able to provide indexing on the individual nodes. Okay, fine. So I'll skip caching, materialize view, so this is the same thing as logical data dependence except you can actually uh pre-generate the views as opposed to evaluate them all at runtime, but we're not too much about that. And then transactions, which I'll talk about in a couple of segments in the context of of nosql but while databases. So database is very good at transactions; they were thrown out the window among among other things in this kind of context of mapreduce and nosql, and they're starting to come back, but remember, you know, what mapreduce did provide was very very high scalability, so this is, you know, thousands and UPS a thousand machines and up, and it also provided this notion of fault tolerance. So relational databases didn't unders didn't really treat fault tolerance this way; they were unbelievably good at, you know, recovery, right? If you were because of this notion of transactions, if you were sort of operating on the database and everything went put uh given some time it would figure everything out and recover uh and you would you you can be guaranteed to have lost no data. Okay, that's fine, but that's not the same thing as saying during query processing while a single query is running, what if something goes wrong? Do I always have to start back over from the beginning or not? And sort of the implicit assumption assumption with relational databases was that your queries aren't taking long enough for that to really matter, but in the era of Big Data, of massive data analytics, of course you have queries that are running for many many hours, right? And so having to restart those, and of course they're they're running on many many machines where failures are bound to happen, and so that context uh is something that mapreduce sort of really motivated, and now you're you're you see modern parallel databases capturing some notion of fault tolerance uh in general. Okay, so these are sort of a list of some kind of partial list of contributions for relational databases, and this is a partial list of contributions, well maybe maybe a complete list of contributions from appreduce, and my point is that you see a lot of mixing and matching going on; the design space is being more fully explored. It used to be sort of all about relational databases with their their choice in the design space, and then mapreduce kind of rebooted that a little bit, and now you see kind of a more fluid mix; people started cherry and picking features. Okay, fine. And the last one I guess I didn't talk about here is what I think was really really powerful about mapreduce is it turned, you know, it turned it turned the army of java programmers that are out there into distributed systems programmers, right? A mere mortal Java programmer could all of a sudden be productive processing hundreds of terabytes without necessarily having to learn anything about distributed systems; that was really really powerful, right? The analog in databases was I mean you had to become a database expert to be able to use these things. Okay, and so I think that that impact is hard to overstate, right? The ability for one person to get work done that used to require a massive team in six months of work was significant. All right [Music] Okay, so just to wrap up this discussion of mapreduce versus databases, I want to go over some results from a paper in 2009 that's on the reading list where they directly compared a Hadoop and a couple of different databases and see if we can maybe explain what some of these results tell us. Okay, so this was uh Andy Pavlo and some other folks at MIT in Brown who did an experiment with this kind of a setup. So the comparison was between three systems: Hadoop, Vertica, a uh which was a column oriented database, and dbms X, which shall remain unnamed, although you might be able to figure it out. And so we haven't learned what a column orange database is and what a row range database is, but we may have a guest lecture later that will describe that in more detail, but for right now for the purposes just think of these as two different kinds of relational database or two different relational databases with different techniques under the hoods under the hood. Okay. And so there's two different facets to the analysis: one was sort of qualitative about their discussion around the programming model and the ease of setup and so on, and the other was quantitative, which was performance experiments for particular types of queries. Okay.

So the first task they considered was what they call a grep task, and so this is uh a task to find a three byte pattern in a hundred byte record, and the data set was a very very large set of Hunter byte records. So this was done in the this task was performed in the original mapreduce paper in 2004, which makes it a good candidate for a benchmark. And so the data data set here is 10 billion records with, you know, totaling one terabyte spread across either 25, 50, or 100 nodes. Okay, so you're just trying to find this record, so this is much like this, you know, genetic sequence DNA search task that we described as a motivating example for sort of describing uh scalability. Okay, fine. So what were the results? Just to load this data in, this is what the story sort of look like: Hadoop and the system called Vertica that they're really the theme Here is they were the designers of the Vertica system, and so most of these results are going to show Vertica doing quite well for a variety of reasons. So we're not going to talk too much about those particular reasons; we're mostly going to be thinking about dbmsx, which is a conventional relational database and Hadoop. Okay. So here loading is fast on Hadoop while loading is slow on the DB on the relational database, and again it was sort of fast on Vertica as well. So why is it faster on Hadoop? Well, there's not much to the Loading, right? You have to put it into this hdfs system, so it needs to be partitioned, but that's about it. When you put things into a database, it's actually re-casting the data from its raw form into internal structures in the database, and that takes time. Okay, and the process could be even worse because if you're building indexes over the data, you actually, you know, every time you insert data into the index it needs to sort of maintain that data structure. Okay, and so load times are known to be bad. So the takeaway here is remember that load times are typically bad in relational databases relative to Hadoop because it has to do more work. Now once it's in the database, you actually get some benefit from that, and we'll see that in a second and even in these results, but it's actually, you know, we know that it conforms to a schema, for example. Hadoop, it's just a pile of bits; we don't know anything until we actually run a mapreduce task on it. Okay. And so how much faster? Well, in their experiments uh for the on 25 machines, you know, we're up here at 25,000; these are all seconds by the way um, you know, 7,500 seconds versus 25,000 and a little bit less as we go to more servers. Okay. Now actually running the grep cast to find things uh this is what we see again, maybe ignoring Vertica for now because I haven't explained to you what, you know, what the difference about Vertica is that allows it to be so fast, but just think about a relational database from what we do understand [Music] Hadoop is slower here, and the primary reason is that it doesn't have access to an index to search. Okay, so again, no index is available; Hadoop has to do that's wrong. Okay, so Hadoop is slower than the database even though both are doing a full scan of the data. The grep task here is not something amenable to any sort of indexing; you actually have to touch every record, so there's no fundamental reason why the database should be slower or faster, but partially because it gets a win out of these structured internal representation of the data and doesn't have to uh re-parse the raw data from disk like Hadoop does. And so I said there's no fundamental reason; there is a fundamental reason because it's already in a kind of a packed internal binary representation which we paid for in the loading phase, but now we get the benefit from here in the query phase even before we even talk about indexes. Okay. Now a selection task where you're not having to scan every record necessarily, you know, that is immutable to indexing as we discussed in the scalability segment, well, the story is even, you know, more extreme, right? The Hadoop results are just way way way higher than both the database and in particular uh the the Vertica results. And so here the reason is because you can build an index on the page rank attribute and zoom in directly to the records that you're interested in. Okay, fine. So those are sort of search and retrieval tasks, not arguably not exactly what Hadoop was designed for; it was designed more for analytical tasks. So now consider these. So here the data set is 600,000 HTML documents, which works out to be about six gigabytes of data per node, along with another data set is 105,155 million user visit records and 18 million rankings records. So this is kind of a web data processing task. All right. And so a simple aggregate task here is to add up all the ad Revenue corresponding to a particular subdomain. So they apply this function substring to the source IP to pull out the first seven uh six characters, right, the subnet mask of the first prefix of the of the IP address, grouped by that and just add up the ad Revenue. Okay. So this is one thing this point to point out is it's actually very nicely and simply expressed as a SQL query; you don't necessarily have to write a bunch of java code in mapreduce to express it in this particular case. Okay. And so here are the results; again, you see a row range of database beating uh Hadoop, and the reason here is maybe not quite so easy to explain, but essentially it's the the internal representation in the database again wins; there's no parsing that has to happen. Okay, okay. On the same schema, there's a join task uh which is to find the source IP that generated the most ad Revenue along with its average page rank. And so this is kind of a complicated thing involving multi-step multiple passes over the data, you know, to sort of compute the average pay page Rank and then find the source IP that find the maximum ad Revenue, find that corresponding Source IP, and then compute its average page rankings. So the implementations here are a fairly complex SQL statement involving the use of temporary tables, and in mapreduce it has to be three separate mapreduce jobs chained together. Okay. And so for the complicated SQL, we won't go through this in too much detail, but just notice that there's a join and then there's a group by, you know, we looked at some complicated SQL and we sort of showed how to break them down, and this is no different, so there's a, you know, you know, see you see two tables which you should think to yourself join, and then you see a group buying, so those are really the two tasks going on, and then the second step is to um do a big sort because you see the order by and just find the topmost record. Fine, a joined in a group by. And so here are the results are also pretty impressive, and the reason is again because of the indexing, right? This join can be done very very quickly because there's different kinds of ways to do the join; the one we described from approduce is when you have no information about the schema; all you've got are these two big relations, and you have to scan them both in parallel and shuffle them across the network with respect to the join key and then perform the join, but if one of them is indexed on that join attribute, you have other plans available to you, and the database is automatically going to figure out the right one thanks to the magic of relational algebra. And so that's what's going on here, and so both Vertica and the relational database can do a lot better. Okay. Now so that's fine, so that sort of Paints the picture that maybe relational databases are are great, and boy this this, you know, mapreduce framework is is all wet, you know, and why would anyone use it? Well, we talked about fault tolerance uh but a couple of other things, you know, there's other ways to avoid sequential scans that you can actually Implement directly in Hadoop. So for example, if you have a large relation in a small relation, one thing you can and the relation

is small in the sense that it that it fits and fits on a single node. It doesn't need to be partitioned anymore, which happens a fair amount. You can actually broadcast that, make a copy of it, and send it to every machine in the cluster, every—at least every machine that has a has a copy of the other relation you're joining against. Right? So you're joining R and S, and S is small and R is big, we'll just copy S to every partition of R, and now you can do the join locally without having to do this sort of shuffle phase. Okay.

And so they didn't get a chance to take advantage of um that mechanism. Moreover, there are—especially in modern systems—this paper was in 2009, which is now a little bit old, or quite a bit old—there are ways to to provide indexing capabilities in a Hadoop stack, and so you're not sort of dead in the water when you aren't allowed to use indexing. So the positive view, you know, if you look sort of warmly on this work, you can think, uh, great, you know, relational databases have all these benefits, and Hadoop can't really compete on some of these—they're even very basic queries. Another way of looking at this as well: these tricks that we already know work really well, like indexing, do indeed work, and so all we got to do is add those to Hadoop, and we'll get the same kind of benefits. Okay.

So what's interesting here is to read about the response from Google when this paper came out, which was a discussion publishing in CACM. And one of their points was that the largest known database installations were both at eBay at the time, which was a Greenplum on about 100 nodes and a Teradata system on also about 100 nodes. And the largest MapReduce installations at the time were way, way larger—right, nearly four thousand at Yahoo and 600 plus at Facebook. And again, this is years, years ago, so these numbers are much higher actually in both cases, but I think the overall point is still the same: the the size of uh, even perhaps typical Hadoop cluster is pretty enormous. Okay.

To conclude the comparisons, we've said this a couple of times, but just to wrap it up one more time: what can MapReduce learn from databases? In the words of the authors of this paper, is that declarative languages are a good thing; schemas are important. And what can databases learn from MapReduce? Is this query-level fault tolerance; support for what I'm calling is in-situ data, which is, you know, data as it lies, right; supporting you without—don't require that the data be sort of transformed and loaded before you can work with it; and then maybe embrace open source. Because if they had—if, again, if there had been an open source parallel database available, you might not see the same popularity in in uh MapReduce. Okay.

Other systems that have been considered in the same kind of framework after the fact were Hadoop DB, which became Adapt, which I mentioned is now a startup. And this is Hadoop file system but has Post—it's actually not Postgres anymore; it's um, ADB—but the point being a relational database on individual nodes in order to get some of the indexing, of some of the at least local level benefits of relational query optimization. And then Hive also came out since then. Fine. So that's the end of MapReduce—com—of both MapReduce itself and MapReduce compared to relational databases. And in the next segment, we'll talk about NoSQL systems that are solving a slightly different problem.

[Music] Talking about NoSQL, and so these systems are typically associated with building very large scalable web applications as opposed to analyzing data, which is really the focus of this course. However, I think it's important to cover this topic for a few reasons. One is, you know, as a data scientist, you'll be manipulating data that is increasingly found in one of these variants of a NoSQL system. But but also the the systems and the terminology in this space are really influencing people's thinking about how to deal with large-scale data. And so being cognizant of the landscape here and being cognizant of the major trends in the history allow you to make informed decisions about this, you know, as a data scientist, you may be asked to make recommendations about what kind of platform to go with to do your kind of—to do your analysis. And so understanding the pros and cons of limitedisms and how they work can be pretty important. Okay.

You know, and then maybe third, the same concepts that we've been discussing in other segments, you know, sort of relational algebra, logical data independence, you know, simple analytics going a long way, simple, simple scalable analytics, uh, indexing—these ideas come up in this space as well, and so it's yet another application of these concepts. And then maybe finally, the data scientist may be called upon to actually build some of the large-scale systems as part of their work. It's not unheard of, you know, I mentioned in the first few segments that building data products was an important part of data science, and so one—that one manifestation of data products is some of these large-scale web applications. Okay. And so NoSQL systems may or may not be a part of that. Fine. So let's get started. All right.

So where we are so far is we've mentioned that data science maybe has these three steps: data preparation at large scale, all right. And we talked about manipulation and munging and data Jiu Jitsu and so on, and that's all this step one. And then step two is analytics, or actually running the model, and we're going to get that in the next week. And then third is communicating the results, interpreting and communicating the results, and this will be a visualization will be a big part of the way we talk about communication. Okay. So we're still in the this data preparation phase, and we're spending a little bit longer on that than, you know, say a third of the course—more than a third of the course—for the reason that I gave in the first few uh segments, that really this is the part that keeps people up at night. So I want to make sure that you're armed with how to use databases to do this data managing, how to use MapReduce to do this data munging, uh, also make the point that a lot of times even the analytics itself can be pushed into these systems, which you saw in hopefully in assignment two in the database assignment. Okay.

So then some of the key ideas from databases that we took away were this concept of the relational algebra—comes up even extern—even outside of databases—it's not only found, you know, with SQL systems, and we'll see that again—and this notion of physical logical data independence comes up over and over again—maybe indexing—and then we talk about MapReduce, and we gave a lot of examples of some basic operations of MapReduce, and we started an assignment involving writing your own MapReduce programs, at least at the programming model level. And so here we saw that part of the advantages of the system itself—firstly Google, and then an implementation Hadoop—was fault tolerance at scale, and another one was you didn't have to load the data; you could just work with it as is, unlike databases where you have to actually impose some sort of a scheme on it in order to even touch it for the first time. And so that first step can be a doozy. Okay.

Extending this point, your direct programming on in-situ data—so anything that comes at you, you can sort of—if you can write a program that can process it, you can probably write a MapReduce program they can process that scale, right? And that's very, very powerful thing. All right. Maybe another way to put this is sort of single developer—right, you're kind of up and running within the hour with MapReduce, and that was never really a property that databases had; it was always a sort of difficult—it was a significant project to get a database installed and running. Fine. So that's the background, but we haven't talked about is this NoSQL system, and so I want to use this table as a way to sort of organize the roadmap for this discussion. And so what I've done here is tried to list out, by feature, a bunch of relevant systems in this space, and right now they're sort of sorted by time. And so these features are admittedly selected by me for what the important ones are, but I don't think they'd be too controversial, and I don't think there's anything obviously missing here either. Okay.

Going through these briefly then: a major one here was that it needs to scale to sort of thousands of machines, and there's some need to look up by a primary index—what I mean by that is by some key value—right, you can look up by some record identifier. Okay. And then another feature that they may or may not have is the look up by secondary indexes—so what I mean here is you can look up by some attribute that is not that key. Okay. And so databases have this—for example, you can build an index on any attribute you want, and the optimizer will take advantage of it. Uh, the third one—and this, this is the one we'll spend a lot of time on because it's sort of pretty fundamental to the motivation for why NoSQL systems sort of earn their own moniker or are different—are transactions. Okay. And you can also see that there's some complexity here that I'll try to explain as we go through this, so it's not just a yes or no; it's kind of depends on the case-by-case. Okay. And then this field is whether or not these systems support essentially joins, but I've generalized that to analytics, and I'm perhaps guilty of making those two things almost synonymous—if you—if you can do some—if you can do joins, then there's a whole space of different kind of analytics things you can do, and if you can't do joins, then you're leaving all that work up to the client, and all you can do is retrieve data. And so that's really a a a key indicator of how much computation you can push into the system and how much you have to sort of bring the data back out to the client. Okay. And then integrity constraints, and I debated about calling this schema instead of integrity constraints, but as I think we'll see, uh, arguably some of these systems do indeed have a schema but may or may not actually enforce that schema, and so it's just to avoid the confusion I'm going to call this integrityune—so this is sort of hard schemas, if you will. Okay. And then views, which if you remember is—is I'm declaring that to be synonymous with logical data independence. And then finally, is there some sort of declarative language or algebraic way of programming against this thing, or is it really just sort of a a low-level simple operation API? All right.

A couple of caveats here: one is I haven't included any parallel databases on this list at all, although there's absolutely no reason why you couldn't include them here—they would tend to have a lot of check marks across this row—um, but since the focus is NoSQL systems, I am leaving those out. And then the other caveat I make is that, you know, individual cells in this table may be debatable depending on how you interpret the the column, so this isn't necessarily meant to be hard and fast rules, but again, I don't think they'll be wildly controversial either. Okay.

So one of the first stories you can tell by staring at this table is that relational databases, you know, been around for quite a while and have check marks sort of everywhere—have all these features—except they weren't really ever shown to scale to lots and lots and lots of computers, right? Everything was sort on the order of tens of machines. Okay. So why don't they scale well? We talked in the MapReduce segment about sort of read performance and related and sort of analyze some experimental results from a paper in 2009 comparing sort of the benefits and strengths of MapReduce versus uh databases. So maybe on read performance there's an argument to be made that they did scale, but one—one area where they certainly didn't, or at least certainly weren't shown to scale to this level, was in uh updates. Okay. So not just the read and not just the analytic workload, but the transaction processing workloads. All right.

[Music] Okay. So we start with the same schematic that we were looking at—we were talking about MapReduce and scalability, where we take a big data set and break it into chunks and send those chunks to different machines. Okay. And here we are replicating this chunk to three different machines, which is the same thing we did for the Hadoop file system for fault tolerance purposes, where, you know, if this machine dies, we still have two copies of the data to draw from, and we do this with every chunk. All right. But the two questions we—the two requirements we need to speak to here is we need to ensure high availability so that when something goes wrong uh the data is still available, and we also want to support updates in this context, which is different than we were talking about before. So instead of just read performance or fault tolerance in the context of reads, we also want to make changes to this data now and have them propagate to both other replicas and in some cases to other uh consumers of that change—right, there might be other blocks of data that that are refer to the same information, and I'll give you an example in the next slide. Okay.

So imagine a social networking application where people are updating their status, and their friends get to—you know, your friends get to see your status updates. Okay. And so the right operation here is Sue updates her own status, and the question we ask is of her friends—what happens? Who sees the new one? Who sees the old one? How do these—how does this status change propagate? And they answered this question from a database perspective was, well, look, you know, everyone must see the new change or no one does—right, either the transaction commits and all copies of the data everywhere are synchronized simultaneously, and further anybody attempting to read the value in a new media state is able to either read only the old value or is—or has to wait until the transaction commits, right, which could be an arbitrary—a pretty long time. Deadlocks can happen is why I said arbitrarily. Fine. So that's the answer given by databases—everything's synchronous, everything must be updated; it's either all or nothing. Okay. And the NoSQL system sort of made this observation that said, well, look, for really large applications, we simply can't afford to wait arbitrarily long for this to happen, right? I mean, you need status updates to be able to commit and respond so the user can go on and do other things, right? They can't sort of look at an hourglass while uh the synchronization is still occurring. You know. And then further the observation is, well, maybe it doesn't matter anyway. I mean, is it really that important that, you know, here—if—if Sue's friend uh Joe sees the new status while Kai still sees the old status—maybe who cares, right? As long as Kai eventually sees the new status, maybe that's good enough. Okay. And so these observations suggested moving in a different area of the design space in sort of high scalability, high availability, and uh consistency—application consistency—and that motivated—and and those that space of systems started be associated with kind of anti-database—right, it took a very different approach than databases did, and so the term NoSQL came into play. It's actually unfortunate that the uh, you know, the name that stuck was NoSQL because it doesn't have all that much to do with SQL, right? It has more to do with the transaction processing side of databases, which is not all that relevant for SQL, right? I mean, the model of transactions, the sequences of reads and writes—nothing to do with a query language—but hey, that's what stuck. Now, I—I don't mean to say that the term NoSQL only suggests these transaction models; it also sort of suggests a weaker data model and so on, and we'll talk a little bit about that, but I—I want this point to come across because this is one of the key ideas. Okay.

How do databases solve this problem, or why do they take so long? Well, there's a protocol called two-phase commit that's fairly standard in these situations for synchronous processing. And so the motivation for why you want two-phase commit goes like this: if you want to have a bunch of replicas or other kinds of subordinates—anybody that wants to see the change—you know, your—the—you make your status update, and your friends need to see it—the server's holding on those different friends need to be told of the change, and so if you just go ahead and tell them—say, look, I made this change, go ahead and update your internal state to reflect Sue's new status—then you can have—you know, some of them report back success, but one of them could fail, but now you're in trouble, right? Because this one has the old value because it failed for some reason—either you didn't hear back from the server at all, or it said, look, something's wrong with my disk; I can't do it—so it responds with a failure regardless, and these two have already successfully applied the—I'll put a check mark—I've already successfully applied the transaction, and so now you're in an inconsistent state—subordinate three has the old value, and these guys have the new values. Okay. So how you solve this problem is two-phase commit. And so the first phase here is the coordinator sends a prepare-to-commit message, and the subordinates make sure they take action to make sure that they can uh commit that transaction when asked, no matter what. And so typically this means writing to a log the the information related to the transaction so that even if the power goes out, when they wake back up they can pull it from the log. Okay. And then subordinates reply with a yes, I'm ready to commit. And then in phase two, if all the subordinates say they're ready, then you'll go ahead and send the commit message, and if anyone failed—if rather—instead anyone failed, then you send back an abort message, and the individuals can clean up. Okay. So this is fine, uh, and here's a schematic of it: in step one, they say prepare—these guys—all right ahead to the log and say I'm about to write this transaction, or I'm going to commit this transaction—they respond with yes, I'm ready to do so—the coordinator comes back with commit—and then finally all the work is—is done. And I'm not going to show the schematic for what happens in a failure, but essentially the coordinator needs to watch out for it and send back an abort if if something has gone wrong. Okay.

Okay, so there's a couple of problems with this: one is there's some dependencies on the coordinator here that if the coordinator fails at the wrong time, things can go kind of screwy. And a fully distributed protocol for ensuring mutual commitment of transactions or other kinds of operations can be achieved, and uh one of the most successful and popular methods of doing this is an algorithm called Paxos that we're not going to talk about in detail, but you're going to see that term if you look at some of the readings for the NoSQL systems. Okay. So think two-phase commit on a local cluster for a database; think Paxos for a distributed sort of peer-to-peer kind of protocol. And just briefly what Paxos is essentially doing is it's a voting scheme, so people sort of vote on—you know, the individual servers will have to self-determine whether or not they're supposed to commit the transaction or not—and at the details can get a little bit subtle, but but overall it's pretty simple, given—given the nature—given the difficulty the task is involved. Okay. So fine. So that's one problem; the other problem is just that with impacts of sort of shares is that this can take a while, right? If subordinates don't respond promptly, you might be waiting around; if things fail multiple times, you need to sort of abort and retry transactions at the application layer—things can go slow uh when there's—it doesn't necessarily scale when there's thousands or millions of subordinates that need to do this—you're kind of dead in the water. So other protocols that I'm not going to talk about in too much detail include a multiversion—but you will see in some of the papers mentioned multiversion concurrency control, where each right creates a new version of the data item, and the legality of a read is determined by checking the timestamp of the read transaction versus the current timestamp of the version that you're trying to read. Okay. And if it's been updated since the time you're supposed to be reading it, then uh, you know, prior to MVCC, all you could do is abort the—abort the read and say, look, you're looking

At dirty data, you're done. But with multiversion concurrency control, you can actually keep multiple, multiple versions around and redirect the read to the potentially to the prior version that is correct. Okay? And thereby avoid, avoid, avoiding, avoiding certain transactions. Fine. So that mechanism still has a dependency on a coordinator role to administer timestamps. A fully distributed scheme where the decision to go forward or the transaction or to abort a transaction is made through a voting scheme among peers is Paxos. And Paxos is very successful and very widely applied, and you'll see it mentioned in some of the NoSQL papers if you take the time to read them, and they're on the reading list. And so this relieves the dependency on having a central coordinator, but is still synchronous and still has the potential for deadlock and can take some amount of time to reach consensus depending on what's going on and what kind of failures are happening. And so it's difficult to guarantee very high performance, a very low latency response times. All right.

So then the term eventual consistency was originally defined not so much in the context of its utility and allowing systems to scale to very large levels, but just in this argument that the right, the only uh players in distributed systems that could make the appropriate decision about how to handle conflicts were the applications themselves. And so it was a version of this end-to-end argument that you may or may not have come across in the context of networking. And so this was a paper in 1995 by Doug Terry where this term was coined. And so he says, you know, we believe that applications must be aware that they may have read weakly consistent data and that their write operations may conflict with those of other users and applications, and that applications must be involved in the detection and resolution of conflicts, and studies naturally dependent on the semantics application. And so I will make the argument in a few, couple of segments that I'm not sure I totally agree with these assertions, that it actually is better for the system to take care of this when it can, but what I want to do is let you know that this is where the term comes from as opposed to the NoSQL system in the last 10 years or so, which is really what, where it increased in popularity. Okay.

So what does it mean? Well, what it means is in the absence of updates, all replicas will eventually converge towards identical copies. Right? So as long as things don't continuously change, as changes settle down, we'll all eventually see the same value. Right? All your friends will see your status. Right? They won't be permanently stuck looking at an old one. But you know what the application sees in the meantime, what's one of your friends, which which status one of your friends might be looking at, is really sensitive to the internal details of whatever application you're building and as difficult and is therefore difficult to predict. Okay. And so for this reason, it's, it's a little bit difficult to reason very precisely or formally about what eventual consistency means because it is so dependent on particular implementation details that are themselves difficult to formalize. Okay. And in general, contrast this with what we've been talking about, relational databases and things like Paxos, where they guarantee strong consistency, but there may be deadlocks. And so it's, you can prove that no system can be free of deadlocks and guarantee consistency. And so relational databases and Paxos give up on this liveness property, meaning that they, they might allow deadlocks in favor of strong consistency. Now they've, you can show that the cases where deadlocks can occur can be made sort of rare through different design decisions, but they can still happen. Okay. Fine.

And so visually consistent models say we can't afford the cost of waiting for these protocols to run, and moreover, they may not be necessary in certain application contexts. All right. So where we are now is we're looking at this column, and I've already sort of marked this up a little bit, but the what these words now mean, uh, and we'll, we'll talk about these a little bit more when we talk about a few of these systems, is the scope of where strongly consistent transactions are supported. And so the scope here of a single record means that I can update multiple fields in one record, and either all the change will occur, changes will occur, or none of them will occur. Okay. Uh, by the way, I filtered this list to only include NoSQL systems, so relational databases support this across arbitrary records. Right? You can have, you can update a record over here, and you can update a record over there and call that one transaction, and the system will only see both of those changes or neither of those changes, and that's what's not supported with these NoSQL systems. So within individual record is supported within some of these systems; nothing's even supported. All right, you keep, you know, there's no guarantees at all, really. And what this EC means is eventually consistent, so it's not really strongly consistent; it's not, it's just not a transaction, but they do have eventually consistent guarantees at the record level, and that's what all these systems sort of guarantee. And then this system, MegaStore, that's based on Bigtable from Google, it's also a Google system, uh, defines a notion of entity groups, and this is a set of related records for which transactions are strongly consistent for that group. Okay. So this is a little bit better than just one individual record is the only, only guarantee we can give, and it's a little bit less than any arbitrary record in the database; it's predefined entity groups that allow transactions. Okay. So this is sort of a compromise. Fine. And then this most recent system from Google, Spanner, offers true strong consistency across all the records, and we'll talk about why they made that choice in a little bit. Okay.

So another concept I want you to be familiar with is this so-called CAP theorem from Eric Brewer in 2000, follow-up paper and, uh, by Lynch in 2002, where they define these three notions: consistency, availability, and partitioning. And the way this is often described is you have to choose two of these, you know, you can't get all three, to choose two or sacrifice, uh, performance. But I don't really like thinking of it that way, and Eric Brewer has also sort of described that maybe that's not the right way to think about it. And the reason is because it's not clear what it means to choose consistency and availability at the expense of partitioning. Okay. So what is partitioning? Partitioning means, well, if you've got a big distributed system with hundreds of nodes involved, hundreds of servers all communicating with each other, and some segment of them lose communication with the other servers, can those two segments still make forward progress in the application independently and sync up later, or does everything have to stop and wait, or certain nodes have to stop and wait in order to re-establish communication? So, for example, if you have a master node that controls everything and you have some worker nodes that lose contact with the master, there are many designs at which you can't make any forward progress until you re-establish connections with the master node. Right? You can, you can do no useful work because you're waiting on communication; you're waiting on that last message from the, from the master to tell you what to do next. Uh, so in those cases, you've given up availability. Right? You go down; those nodes are no longer accessible or doing useful work in the context of a network partitioning. Okay. On the other hand, if you do say, well, sure, we're going to continue doing useful work even independently, then it's not difficult to show that you can arrive at an inconsistent state. Right? Updates are coming into this partition, and updates are coming this partition of the network, and sometime down the road, communications re-established, you find out, oops, you know, your replica has one value, my replica has another; which one's right? Well, we're going to sort it out, but meanwhile, we've already sort of exposed these values to the application, so in some sense, we're demonstrably inconsistent. Okay. And so the point is you can't get all three of these. All right. So you've either sacrificed availability or you've sacrificed consistency by allowing things to continue working. So conventional databases essentially assume that there is no partitioning, and again, this, this is a function of them only operating on tens of nodes at a time. Right? They didn't go to this sort of thousand-node scale or planet-wide distributed, uh, systems, uh, and so you could kind of assume that there wasn't, there wasn't really a need to worry about, well, what if queries are coming into half my nodes and they can't talk to the other half of my nodes, and so on? They're all sitting there in a cluster that's in your, in your data center or not even your data center, in your server room to some extent. And so that was really was an issue that they were thinking about too much. Okay. And the NoSQL systems do need to worry about this; they are very large, they are very distributed, there are different kinds of Byzantine failures happening all the time because, because of the sheer scale, and therefore they choose to sacrifice consistency instead of availability. Okay. And so graphically, you can look at this in this sort of triangle forming; you can put different systems on sort of an edge here where relational databases assume consistency and availability, uh, but, but assume partitioning can never happen, while other systems need to tolerate partitioning but give up on availability, uh, and they are ensuring that certain kinds of transactions are going to be consistent. Okay. And then other systems say, well, we're going to give up on consistency, but you can always do useful work. Okay. So fine. And really, this is the important thing here is the, the scope of the transaction that I put in that table is sort of critical here too; it's not sort of nothing except for Spanner over here even tries to provide global transactions like relational databases do. Okay. All right. So fine. So I'll pick up here in the next segment [Music].

So Rick could tell, wrote a nice paper in 2010 about scalable SQL and NoSQL data storage where he had a taxonomy of these systems and placed popular instances of these systems into that taxonomy. And so this slide corresponds to his grouping where each color is one of the groups that he defines. So he's already lumped them into key-value stores, document stores, and extensible record stores. And so when you went in by that was a document is, you know, an example; this is an XML document or the JSON object that we looked at in the Twitter assignment where you can have sort of arbitrary, arbitrary nesting, and it's also extensible; you can add new things to it whenever you want. Okay. So no top-down schema being enforced. Okay. An extensible record you can think of is much like a database record except that new attributes can be added. Okay. So there is some notion of a schema that's used for different various purposes; in particular, there's sort of groups of attributes that are manipulated together, these families, but you can stick new attributes on an individual row, which you're not allowed to do with a relational database. And then finally, a key-value, I'm using the term object here, is a set of key-value pairs. So the difference here is that there's typically not a schema of any kind, so you don't care what keys there are; it could be any keys at all, and there's no exposed nesting. And what I mean by that is, you know, a value can be anything you want, so you might have some kind of complex object in the value, but it's not going to be sort of visible to the system; it's just going to be a blob, a black box object that, that the system doesn't know anything about. Okay. So sort of only one layer of nesting is aware of the system unlike a document store or a document object that might have multiple layers, nothing that are exposed to the system. Okay. And so his characterization of NoSQL features, you know, admitting that there's perhaps not a formal definition of, well, there certainly isn't a formal definition of NoSQL, but the term tends to be applied in context of systems that have these features. So the sum ability to scale simple operation throughput to many, many servers, and by simple operation here we mean key lookups or maybe even attribute lookups or reads and writes of just one or a few records. Right? So these sort of needle-in-the-haystack kind of operations as opposed to these big analytic queries like we've been talking about with MapReduce and with databases. Okay. And second criteria is the ability to replicate and partition data over many servers here. So, you know, break a single large data set into multiple pieces to automatically and, you know, to manage this yourself. And here you might see the term sharding and horizontal partitioning; the difference between the two, if there is any, is not particularly important, so you can think of them as synonyms. Whenever you see sharding, think horizontal partitioning of a, of a database table. You'll see the term horizontal partition you use more in the database community and sharding use more in the NoSQL community. Okay. And then a simple API, so no query language, uh, just corresponds to the first poll; these are the simple operation operations. And then critically, a weaker concurrency model than what I'm saying is ACID transactions now, and I'm like, we'll go, we'll talk a bit about ACID in the next slide; I'm not going to go into a lot of detail here. There's, you know, 40 years of research on this topic, too much to cover in this course, especially when we're mostly focused on, you know, reading data, analyzing data as opposed to a concurrency control, but we will spend some time on some techniques of concurrency control in a minute. Okay. Uh, and then some of efficient use of distributed, he talks about efficient use of distributed indexes and RAM for data storage. So this is kind of minimizing latency is their emphasis as opposed to just throughput. Right? And then you typically have this ability to add new attributes to data records in various ways as we talked about on the previous slide. Right? So the, the lack of a no schema is what you can think of here. All right. So this is no schema, no transactions, and we'll go into more detail there; no query language with NoSQL. Right? And high scale. Okay.

So ACID, and he talks about this term BASE that never quite caught on; I, I wouldn't typically use that this term, and I'm not sure I recommend you do either. Certainly ACID is much more permanent in the vernacular than BASE is; it's, it's okay. So ACID is an acronym standing for these four concepts: atomicity, consistency, isolation, and durability. And just briefly, this is, you know, the context here is when we're modifying records in, let's say, a database, and we can be modifying lots of different records across different tables, anything we want, and the point is they're all lumped into one transaction. And so each one of these refers to, you know, that's the context we're in for each one of these concepts. So atomicity means that the entire transaction either needs to succeed or needs to fail; you aren't allowed to have partial transactions succeed. Consistency is the slipperiest one in my mind, and this quote down here maybe captures that. So it's sort of any data written to the database must be valid according to all defined rules, and the question is, well, where do these defined rules come from? Sometimes they're actually integrity constraints in the database; other times they're just sort of business logic rules perhaps enforced by the application or just assumed by the application. So it's a little bit difficult to say, prove a system is achieves application-level consistency, but that's the goal. The point is if there's only certain allowed states of the database to have, you can't, you should not have a system that allows transactions to put you into an invalid state. Okay? You should only be able to go from working state to working state. Isolation means that while the transaction is occurring, other readers and writers can't, uh, sniff partially completed values. Okay? You know, partially, you can't sniff values or data items before the transaction is complete; they only get final states. And this one is the one most often relaxed in various ways, in part because it's very expensive and also because it's not usually all that critical. And the durability just means that if you report back to the transaction succeeded, it needs to have actually succeeded, meaning it needs to be written out to some kind of non-volatile storage so that it if the power goes out and the machine crashes, you don't say, hey, whoops, you know, that transaction I accepted yesterday or committed yesterday, well, you need to do that again because it didn't, it didn't take. All right. So that's not allowed. So fine. So these, these all make some sense with, you know, with a little bit of slippery notion consistency as I mentioned. And then the pun here is they tried to sort of force an acronym on BASE, and this isn't Riccati, this came from elsewhere, but the idea is, well, it's basically available; there's some notion of soft state, and it's eventually consistent, and we talked about eventual consistency at least an overview in the, uh, previous segment. Fine. That's all I'm going to say about that.

So something else I like about this paper is he sort of says, look, you know, the major impact systems here are these three: this Memcached or memcached D, Dynamo from Amazon, and Bigtable from Google. And the reason he says he's a major impact is you can kind of trace the lineage and show that other systems are basically taking ideas from one of these three early systems. So Memcached is very, very simple, and we'll talk a little bit about one particular technique that it made popular in a minute, but it's essentially just, hey, look, let's load everything into memory, scale it out across many, many machines. Right? And then we'll be able to serve read requests without having to go sort of query the database; we'll just be able to do it directly from memory. And what's also made this very, very popular is you can kind of install it on top of your either scale, scale out or non-scale-out database, and it sort of just works. Right? It just makes things faster, uh, for, for read-heavy workloads. And that was kind of a, a nice thing. So that's an older system, sort of on the scale of 2003, but it's still very widely used and very, very popular, and there's been all sorts of extensions to it. And so we'll talk about the perhaps the most basic version. Amazon's Dynamo paper, which has been somewhat more recently released as a cloud service called DynamoDB, what they did was they didn't invent the concept of conventional consistency, but they did sort of show that if you relax the consistency notion, that will allow you to scale way, way out. Okay. And so data fetch or not be allowed are not guaranteed to be up to date, but updates are guaranteed to be eventually propagated everywhere they need to be. And we give an example of why this was a good idea in the last segment. And then Google's Bigtable that we'll spend some time on, you know, demonstrated that record-oriented storage is could scale to thousands and thousands of machines, and that was something that databases had not shown. Okay. So let's talk about each one of these systems in turn. So Memcached, as he says, the main memory caching service, no persistence, um, the basic version is no replication, meaning that there's not two, there's only one copy of every cached value. So if something goes down, if that goes down, then it's, it's gone. That's okay; it gives us sort of a cache; it's not assumed to be the golden copy of anything. Okay. That being said, there's been many extensions that provide various these various features, including Membase and Membase. So there's a premature system in still in wide use. And an important concept that they adopted in this context was consistent hashing. So I want to explain a little bit about what consistent hashing is so that's one takeaway from, from this lecture. Okay.

So first, just for those of you without necessarily having too much about background in programming, what is hashing? So what is regular hashing? Well, the problem we're looking at here in this kind of high action is a very, very general concept, very fundamental to all of programming, but in the context of what we're doing here, we're trying to assign data keys to a bunch of different servers. And the simplest way you might do this is sort of a round-robin thing. Right? The first key goes to the first server, the second key goes the second server, and you keep going until you run out of servers, and you start back over the first one. And that's implemented by this modulus function. Okay. So each of these data keys is placed somewhere on this, on one of these servers at various points. Fine. That's how hashing works. What's, what's wrong with that? Well, what happens if I want to add more servers to the mix? Right? I want to scale out to, I want to double

The number of servers, well, every existing data key now needs to be re-evaluated. It's a location needs to be re-evaluated by computing K mod to in instead of K mod N, which means every single data item is going to be remapped at once. So every time you want to add a server, you end up having to move all the data that's already in the system, and you're dead in the water. Okay, so what you want is some notion of consistent hashing, where consistent means when I place something somewhere and I add more servers, it's typically going to stay right where it is. And so there's a pretty good trick that's pretty simple to understand for doing this. Okay, so here's how it works.

First key idea is you're going to map the server IDs into the same space as the key values themselves. Okay, so we apply some function that I'm going to leave sort of unspecified and map server one to some point on this circle, and server two some point on this circle, and server three someplace on this circle. And now what that does is divide this space up into three sections. Okay, and now each key that comes around, I also map it into this circle. So this gets key one, and this gives key two, key three, key four, key five, key six, key seven, and so on. Okay, and now this entire region, server one, server two, server three, this entire region is responsible for all of these data keys. Sorry, this server is responsible for all these data keys, and then this server is responsible for all the data keys in this region, and this server is responsible for all the data keys in this region. Okay, and so what's nice about this is now when I add a new server, server ID equals four, well, let's say it comes around and gets stuck right here. Well, that's a bad spot for it for my example. Actually, let's say it comes around right here. Well, you just apply the same rule; it should be a it should be responsible for every key in this region, which means that these two guys need to be moved from server three to server four, right? But you only have to move that one section of data. And so it splits at most sort of K over N data items. All right, so this is a nice trick, and there's all kinds of extensions for, um, supporting replicas. We need to put data items in more than one place. Well, you just sort of hash it to two different places. So if you want to hash the same data under two different places, compute h of, let's say d is the data key, and then also put it to, you know, put in all three places, and you're done.

Okay, so how do we serve requests in this setup? Well, imagine the key space is divided across various servers in the same way we described, and a request comes into the leader that may be elected among the servers or may just be assigned top-down by the system, or could even be assigned randomly. And the naive way of doing this, as well, this server would check to see whether it has the key being requested, and if it doesn't, it would just forward the request onto the next guy. Okay, but this is no good because there could be many, many servers, and this would incur sort of a latency every time you want to do a read. So a better way of doing this is for each server to memorize the locations of other servers in the ring, and which servers it memorizes is like this: so it knows where itself is, it knows a plus two, a plus four, a plus eight, a plus 16, and so on. And what it does is it knows the key range being managed by each one of these servers. Okay, and so what it can do is forward the request to the server that is closest to the key range it's looking for. Okay, and so this takes a logarithmic number of hops. You imagine there's lots of servers here, and keeping all this information straight when new servers come in is still, each server only has to keep track of the logarithmic number of servers as well. So everything sort of ends up being logarithmic to maintain this. Okay [Music] Okay, so Dynamo from Amazon in 2007, which was a paper, and again a few years later, it's been released as a cloud service called DynamoDB. Okay, and so here we're looking at, you know, scales to thousands of nodes. You can look up things by primary index and basically nothing else; it's just a key-value store, just like Memcached. Okay, right. So what are some of the tricks that led to, well, so some key features are that it has—so I'm talking in terms of DynamoDB, which is the implementation you can now go and use and pay for—one of the neat things here is that it offers a service-level agreement on performance. And so at the 99th percentile, you know, they promised to respond within 300 milliseconds for 99.9 percent of its requests. And the reason they do this on the 99th percentile as opposed to some sort of notion of the average, the mean, or the medians, or the mean of the median, is that that would artificially penalize the people who are using it heavily, right? They would get a disproportionate number of, uh, failed requests, right? It'd be easy to satisfy the average by only focusing on the lightweight users, for example.

Okay, so Dynamo, the system, it's a distributed hash table—that's what DHT stands for—and each key is stored at, or sorry, each value is stored down to locations, multiple locations for replication purposes, and it's up to our application factor of N. And so at location K, K plus one, all the way up to K plus N minus one, and they achieve eventual consistency through vector clocks, which I'll describe in the next couple of slides. And so reconciliation of potential conflicts when things are being written, read, and written happens at read time, which is another maybe interesting feature of Dynamo. Okay, so writes never fail, and they cite in the paper that the reason for this is poor customer experience, right? So if you're sort of typing into your Google Doc—well, it's not Amazon—if you're if you're building an application, a web application where you say you update your status on some social networking site, and it comes back with an error message, says sorry, couldn't couldn't commit, you know, somebody else was editing the same, or you're right, you're editing the same status from somewhere else, um, their claim is that that's more disruptive than, uh, getting the, the, you know, the wrong read, which just seems reasonable to me. Okay, all right. And so conflict resolution for many applications may be the most recent write is the one that wins, or you can, uh, actually have the application control this. In some cases, you may even sort of go back and ask the user to resolve the conflict, uh, manually. Okay.

Okay, so the goal with vector clocks is to detect conflicts in a concurrent read-write scenario, but not to necessarily do anything about them automatically. Okay, so in this scheme, every data item is associated with a list of server-timestamp pairs that indicates its version history. And so in this example, some value D was read by a client, and D1 was written back at the server called SX. And so what SX does is append this fact to this vector clock. So in timestamp one, server SX, you know, created a change. Then some other client reads D1 and writes back D2, and, you know, you might want to append to the vector clock both values, but, you know, this change descends from D2 descends from D1; it was handled by the same server, and so you can garbage collect this part of the vector clock, right? So it's the same server with a higher timestamp; you use it; the old version or the older timestamp is not needed anymore. Okay, and since there were no other conflicts to work on. Okay, but now independently, two different clients read D2 and write back different values; one writes back D4, one divided writes back D3, and these two requests were handled by different servers, Sy and SZ, and so these facts get recorded in the vector clock. Since they're different now, the context here—we call this sort of vector clock work—the context will reflect this fact. When the next read comes in, it'll see that, oh, wait, there's a conflict here because we have the same timestamp but two different servers, and you can either ask the client what to do, uh, or you apply some heuristic where the later one runs. These may not be timestamps like integers; they could be sort of actual clock timestamps, in which case you make an arbitrary decision, just pick it and go. Okay, so that's how vector clocks work. So in the example, just to sort of run through, we just saw a client writes D1 to server SX, increases value; another client writes D, reads D1 and writes back D2, also handled by SX, and D1 was garbage collected; then separate clients read D2 and write back D3 and D4 on two different servers, that's Y and SZ; and then another client reads D3 and D4 and notices it and finds that there's a kind of a system reports that there's a conflict to be handled.

Okay, so let's practice with these. Here's two different vector clocks, and you figure out whether there's a con whether they represent a conflict or not. So in this case, we have server SX with the timestamp of three, and on this data server SX with a time period of three, and then each one is a different server with different timestamps. So is there a conflict? Well, yeah, there is, because on one version path, Sy made submit a series of changes, and on another version path, SZ made a series of changes, and they didn't talk to each other because they don't reflect each other's changes. So yes, there is. And then this one, it said the same server at a later timestamp. So is there a conflict here? Well, no, because they weren't handled by different servers. So really, just this one subsumes that one, and we're okay. So on this one, we have server SX with three, server SX with three, server Sy with six, server Sy with six, so they agree so far, and this was an extra change of s of server SZ with a timestamp of two. And so no, there's no conflict here because they agree wherever there's on on this is just a just an extra change on top of this one, so this this guy wins. Okay. And this next one, server SX with the timestamp three, server SX with a timestamp of three, server Sy with a timestamp of 10, and this has server is one of the timestamps is six, and then some later change at SC. So is there a conflict here? Well, there is, because this one is later in time on Sy, right, 10, then this one is, but then, and if that was all there was, if it wasn't for this guy, if it wasn't for, uh, this guy here, will that be okay? We would just pick this one because it's later, but because this one's here, we now have some changes at SZ and some changes in Sy that were both forward in time from the latest point that they agreed on, and so we don't know how to resolve that, and so the yes, there is a conflict. And then similarly here, SX and timestamp three and SX and timestamp three, well, here we have Sy and S1, Sy 10 is the same as the last one, but here instead of six, it's 20, and so it's later than this 10, and then further we have to change it as Z. And so is there a conflict here? Well, no, because this one is strictly later than that one is on all the servers that they share; it has later timestamps, so it strictly subsumes it, and so no, there is no conflict. Okay, so those are in the last time we talked about consistent hashing. In this segment, we talked about vector clocks. These are two little gadgets to be familiar with because they come up a time and again in these NoSQL systems and in other systems in general. Okay, all right.

So Dynamo also talks about a way to parameterize the level of consistency, and this comes up occasionally in papers, so I just want to make sure you're exposed to it. So the idea here is that you have two parameters, R and W, and R is the minimum number of nodes that need to participate in a successful read. Okay, and W is the minimum number of nodes that need to participate in an accessible write. So this is sort of how many replicas you write to and how many replicas need to respond from a pool to know that you sort of have all the information, right? Because everybody's updating everything all the time; it might be that you have 20 different servers; they all have a different version of the data item you're trying to read. And so the question is how many of these do you need to sample before you feel like you have the right one? And so for a replication factor of N, if R plus W is greater than N, then you can claim consistency, but you know, often you want to set R plus W less than N in order to achieve lower latency, so you don't want to have to actually contact, um, you know, too many servers return to in order to satisfy some read request or write request. Okay, so if you see that notation, this is what it means, but I'm not going to describe too much more about it because I think that the sort of, you know, the the formalism falls over a little bit with under a little bit of scrutiny, but but that's what they're talking about when you see this discussion on, say, blog posts. Okay [Music] Okay, so the next system we're going to look at is CouchDB, which began in 2005 but is still has undergone a lot of updates, and it's still pretty popular today. So this is an example of one of these document-oriented stories that Rick could tell talked about. And so here, just to look at the features we're talking about here, we've got the scale, we've got primary index, now now we start to see the requirement for secondary indexes, meaning that you can look up not just by the key but by other kinds of values in the document, and we'll see how that's how that works. Okay, and then transactions also are a little bit better; we can, what I mean by record here is that you can change multiple values within a single document object, which is a set of key-value pairs as opposed to just a single key-value pair. Okay, and then there actually is some support for analytics, and we'll see how this works too. This is all through the notion, this concept of views in CouchDB, but you can actually run little MapReduce scripts to compute, um, derive new values from existing document source. All right, and then the other notice we have here is views, which is somewhat unique. You can see this column is fairly sparsely populated, and I hit the concept of views pretty hard when we're talking about relational databases and argued that it was pretty fundamental to not only the relational model but a pretty important concept in general and gave you this notion of logical data independence. And so whenever you see views, that's a good thing, and so CouchDB has them as well.

All right, so the data model here is document-oriented, we said, where a document is a set of key-value pairs. And so here in this application, you know, these are perhaps blog posts where they have a subject, the subject, you know, the key is subject, and the value is some text string, and they have an author, and they have a date on which they're posted, and then they have a set of tags, and this is sort of what makes it a document model. We said they could be sort of nested, so this is okay to have a list of objects here, and then they have a body, which is another string. Fine. You'll notice that this is actually JSON compliant. We looked at JSON in the Twitter assignment, and so it's a another occurrence of this, and one of the reasons why I wanted to make sure you're working with JSON before is it does get popular. And so CouchDB, all data is represented in JSON, and all the requests coming back are all represented in JSON as well.

Okay, so how do updates work in this context? So as we mentioned, they are, you know, you can make, you train it is fully transactional within within a single document, so full consistency within a document, meaning that I can sort of grab hold of the document and logically sort of lock it and up making whatever updates I want. Now, it doesn't actually take a lock because it uses what's called optimistic, um, concurrency control, meaning that it sort of assumes that the conflicts aren't going to happen optimistically. And so now if I if I check out that document and try to make updates to it anywhere I want all throughout it and then try to commit that change, and someone else is doing this, has done this, has committed their own changes in the meantime, my changes will fail, but we assume that that doesn't happen that often, so it's okay to just sort of fail in those in those cases. And really, wish all you have to do in that case is check out the new changes and make whatever changes you want. Fine. But there's no multi-row transaction. What I mean by multi-row here is multi-document, right? So you can't say, for example, when I update my status, I have to update my document that describes my current state of the world, but maybe I also want to update all my friends' walls, right, their pages to with that reflects my new status, and you can't guarantee that that happens synchronously in CouchDB, but you know, in this particular application and many others, maybe that's okay. All right, you know, you can do it in two separate steps; you up to your status, and then you update theirs, and there'll be a period of time when their, you know, view of your current status is out of date, but maybe that's all right. Fine.

All right, so this notion of views works like this: a view specification, which is itself a CouchDB document, a set of key-value pairs, but a sort of a special one, uh, looks like this. There's some metadata information here, and then there's this key views, which is which is a dictionary of things. And so this this specification has three views in it: one called all, one called by last name, and one called total purchases, and each one of these views is going to be is implementing a set of key-value pairs, right? It's going to be implementing a dictionary. And so the all view. Okay, and so fine. So how are these implemented? Well, you know, speaking of recurring concepts, views are recurring from logical to independence and relational databases, uh, and here MapReduce is recurring even outside of the context of literally, uh, Hadoop. Okay, and so here the map, looks reduced function actually written in JavaScript—again, everything in CouchDB is JavaScript—and they look like this. So the all view has just a map function, no reduce function, and in JavaScript, you can write these anonymous functions like in in this sort of syntax. So this function doesn't have a name; it just says, hey, I've got a function with no name with a single argument called doc, and the body looks like this: it says if doc.type equals customer, then emit a key-value pair where the key is null and the value is doc. So we don't really care about the key in this case; we just care about the doc. All right, so this is all customers. Fine. The by last name view, you can imagine the key is going to be last name in this case, and so here we have another anonymous function: if doc.type equals customer, then emit the last name followed by the entire document. And so now this allows clients to search efficiently by last name, and these views will are intently be your computed, you know, they're materialized sort of eagerly and stored in these distributed beaches or index structures to support various lookups. And so this is how they implement those secondary indexes in that column in our in our table. Okay, so now we can look up my last name as well as by document ID. And so you can even go a little further; you don't have to just do simple key-value pairs; you can even do a little bit of computation. So total purchases here, the map function again takes in a document and says, well, if doc.type equals purchase—so I'm not talking about customers anymore, just purchases—then emit the key of doc.customer and the value of document. But then we're also going to find a reduce function, and the reduce function adds up all the values of the amounts. And so here we have, given a customer, if I give you a customer

You can return; it's the total amount of all of its purchases. Okay. And CouchDB maintains all these views as things change with eventual consistency, sort of semantics. And so we have a lot of things going on here with one concept: we have secondary indexes; we have logical data independence; um, we have MapReduce computation. Okay, so I had checked the box in the table saying that CouchDB could do joins and analytics, and it's not really quite true; they're somewhat limited. So let me show you an example of how they do sort of joins. They have this concept of view collation, and what you can do here is write a map function in a view that looks like this. And so here we're trying to group together all the comments associated with a post, so sort of a join between the comments table, if you will, and the posts and the blog posts table.

And so this map function is the same kind of thing we did in the MapReduce assignment to do a join; this is what you have to do when you do a join. A MapReduce is you take—you pretend like the whole collection of documents, regardless of type, regardless of source, relation—here's one big set of objects. And then your map function, you sort out which one's which and make sure to hash it on the same key. So here, the ID of the document post, and then in this document, doc.post refers to some host ID. And so now you know post ID equals one, and all the comments associated with post ID equals one will go together. But they do this funny thing; they say the key here in the key-value pair is documented, followed by the number zero in the one case and the number one in the other case. And what's happening here is that all everything in Map in CouchDB is sorted, and so you've got things sorted by document ID and then sorted by this extra bit. And so now you have the post ID coming first and then all the comments coming second. And now what could you—you know, you'd think you could just do a reduce function to actually compute the join that you might want, but it turns out reduce functions don't quite work that way, and there's some scalability issues. So what people recommend is to do this view collation trick where you still spit out the data in sorted order, but then you can query it—say, a client application could query it like this—where they say start key is just a post ID without the extra prefix at all, and the end key is post ID with the number two. And so now this gets the full range of the post ID with which has the key of zero followed by all the comments on the post ID of one. So you got everything in one big list, and now you can process it in order to show it—say, show it in the application. So you're essentially constructed the join by cheating with the order. So, so fine. So perhaps you could argue they have some joins and analytics, but it's a big lie to say so. [Music]

Okay, so the third influential system that Rick mentioned in the paper is Bigtable from Google, which is a paper from 2006. And so here we're looking at primary index lookup, secondary index lookup; transactions are also at the sort of scale of the of an individual record; uh, joins, analytics is not supported by being able directly, but in the in HBase, which is the open-source implementation of it, and in—well, in Google's implementation as well—it was sort of designed to be compatible with MapReduce, and so you can run MapReduce on—over here—over the same data that's stored in Bigtable. So they're sort of complementary. And then there's some notion of integrity constraints or schema here, and we'll talk a little bit about how that's implemented. There's no views that I can see, and there's no sort of language level or algebra or algebra level for manipulating these things, or all sort of NoSQL style micro-interactions with individual records and cells. Okay, so this is a paper in OSDI from 2006, and some overlap with the authors of the MapReduce paper, and it was sort of designed from the start to be complementary of MapReduce. So if you remember what was one of the main things that was missing from um MapReduce, or maybe a few things were missing—well, in particular, you couldn't look things up by index; you couldn't get these little sort of low-latency accesses. And so, for example, if you wanted to find all the records, you know, given a big data—say you want to find all the records in some other data set that correspond to it, you want to do some sort of a join—the best you could do is process—you you had to touch every single record; there was no way to zoom in to just the right ones you wanted. Okay. And so Bigtable provides that fast key-based lookup, but you can still process the overall data as a big set of key-value records with MapReduce.

Fine. So the data model here is a sparse, distributed, persistent, multi-dimensional, sorted map. And what they mean here is that you can basically access any cell in a Bigtable by giving a row ID, a column, a column name, and a timestamp. Timestamp isn't really described in this English description here; it's for versioning. So when you have—after—updates, you'll keep track of past versions of the same cell. Okay. And so if you provide these three parameters, Bigtable will return you a string quickly. So each row is—data is all sorted lexicographically by the row key, which is this row ID in this in this bit, right? So this is a some sort of primary key in in sort of relational language, or just a key in kind of NoSQL framework. And then this key range—I say that the instead of integers—then contiguous sub-ranges of this set of keys will be assigned to a tablet. Right. Okay, so this is a little different way of dividing up the data than we've seen in the past in at least one system—we talked about a parallel database model; we happen to use the example from Teradata. And so how did they break up data? Well, they did it by hashing, right? So every individual record would be sent to a server um according to a hash function, which you can generally just think of a sort of round robin. The point is is that two keys that are next to each other in space—so sort of timestamp 5 PM and timestamp 501—there's no reason to believe that 5 o'clock and 501 are going to be on the same server in Teradata's model; here they are. So what are the pros and cons with this? Well, if you're going to typically access a whole range of keys at once, it's pretty nice to be able to, you know, when you get one you get the others too sort of for free because you're you're pulling them all back. However, um, if one particular key range is much more popular than the others—but using the time example again, the most recent data perhaps is the most popular—and so if all the requests are going to that one key range, then you've got a bunch of idle servers hosting all the other tablets that are corresponding to older times, and all the requests are going to this one tablet. And so, for that reason, Teradata sort of chooses to hash everything so that on every request all the servers may have to be accessed, but that's good for scalability. Okay, so pros and cons. All right. So the tablet here is the unit of distribution and load balancing, so they'll move tablets between servers as things start to get unbalanced, right? If you're if your key range is, you know, January, February, March, April, May, and there's a whole lot of data coming in for March, they'll split that into multiple tablets and start and start moving these tablets around between servers in order to balance things. Okay. So within a single table, you can have these groups of columns called column families, and the column names have the family right in there as a qualifier, and this family is the basic unit of access control. So you can provide permissions on a group of columns, memory accounting, and that they're sort of allocated as a unit in memory and in disk accounting, so they moved around on disk as a unit as well. Okay. And so I think this point that typically all columns in a family are the same type, which I find a little unusual because they sort of talk about it being a basic unit of access control, which suggests that there's, you know, things that go together for access controls—sort of social security number and employee ID or something—may or may not be the same type. So there's sort of a logical grouping requirement that they seem to be trying to meet, but then they're at the same time it has to be the same type, which is for very technical reasons—specifically because they want to compress these things. So if you have a whole bunch of integers, it's easier to compress than if you have a mix of integers and strings. Okay, so I think they're trying to kill too many birds with one stone here, right? And then each cell uh can be versioned, which is the third part of that of that key lookup—row ID, column name, and timestamp—and each new version increments that timestamp and say, hey, here you can enact different kinds of policies where you only keep the latest inversions or you keep only diversions since a given a given timestamp. So how these tablets are managed is a master will assign the tablets to tablet servers, and the tablet server handles reads and writes from the tablets it controls. Okay. And so clients communicate directly with a tablet server as opposed to having to go through the master every time, which is helps with scalability. Okay. And then when a tablet starts to get too big, it'll split it and load balance it. All right. So the metadata—keeping track of where tablets are located—is organized itself in another tablet. So there's a root tablet here that describes a a each record in here describes a group of records, a group of location records in a, you know, bigger table. And then each one of these metadata tablets gives the location of a particular user table. Okay. And so this is how you start to keep track hierarchically of where everything is at one time. And so Chubby that they mention uh in the paper is a distributed lock service for controlling access to these things; I'm not going to talk too much about it.

Okay, so how are reads and writes handled in this system? Well, there's a table in memory that stores a sequence of updates as they occur. Okay. And a write operation is low is, you know, adds a record into the memory—memory resident table—but it's also written to a log for fault tolerance purposes. Okay. So if this ever goes down, it reads the tablet logging and can reconstruct what's going on. And then read operations are served uh by reading these SSTable files—you know, the actual data itself—but then also by applying the updates from the Memtable on the fly, right? So it needs—it needs a stream—it says here's the value, and then here's the stream of updates I need to apply to the value to get the true value. Okay. And then there's two—so so this is fine, but but you know what happens when the Memtable gets bigger and bigger and bigger? Well, there's two types of events that occur to, you know, do the bookkeeping here. So one is a minor compaction, and this is when the Memtable gets big—gets written out into an SS—into a new SSTable file, and the changes are merged. Okay. And then a major compaction is—take all the SSTables and rewrite them all into one big one that may be split into multiple files and also clean up any deletes that have occurred. So deletes are just appended as uh instructions but aren't necessarily—it doesn't actually remove anything, so they're sort of garbage collected. Okay. So this way you can keep sort of the read throughput pretty high and sort of keep this upkeep going on in the background. All right. So there's a host of other tricks here too, with a uh—can do various forms of compression—to—specified by—comply—which can be specified by the clients—or some different—different ways of doing it. They use Bloom filters to speed up existence tests. So if you—if you—if I give you a row ID, a column ID, and a timestamp and say find me this value, what these Bloom filters allow you to do are is to very quickly determine whether that does not exist in the system. So these Bloom filter data structures are pretty cool, and I'm going to walk through them in this course in a couple of weeks. Okay. So they help you quickly determine whether that key does not exist in the system; it avoids disk accesses during—during reads. All right. And then there's locality groups you can define, which—another layer of organization on top of families—and these are groups of column families that tend to be accessed together. Fine. And then another trick here is to make sure that the SSTables—these disk chunks—are actually immutable; they never get written directly; the only time they get written is when these major compactions happen and the whole thing is sort of reorganized. And so that means that the only writable data structure is this Memtable, and so the amount of concurrency control to keep things around it remains—remains pretty simple. [Music]

Okay, so Google Bigtable again had a lot of influence, and one of the major outcomes is—just like MapReduce—it was taken up and implemented um as an open-source project called HBase. And so we—where Bigtable is compatible with MapReduce, HBase was compatible with Hadoop. In one slide, there's not much difference here, but I just want to mention the terminology so you've seen it—that there's table at the top level, and then region store, and then Memstore and store file. And so the exact names of these structures are a little bit different, and they did insert one more layer of abstraction, which is this region. Fine. And then how this sort of is compatible with MapReduce is that, you know, each—each one of your map functions will process a single tablet, and so it's sort of one-to-one with the blocks of data that we talked about when we talked about MapReduce. And then I just sort of asked this question—sort of open-ended—is that there's this notion of speculative execution to MapReduce that we talked about whereby, for fault tolerance reasons, it might kick off the same map task twice on two different replicas of the data. And the reason for this is that if one fails, well, you have the other one, right? So you don't have to start over from scratch. But you know, in this environment where you're now working on data that is being actively updated, and that's what HBase and Bigtable are designed to support—is—is updates—it's not quite as clear to me what's going to happen when you have—you know—it's possible, because of eventual consistency, that these two tablets will not always agree instantly on the same record; they'll agree, but different records within the tablet may not. Okay. And so just an example of when you sort of mix these two systems, and there are no sort of system-wide transactional guarantees or system-wide even properties that you can run into—run into trouble. And I think that a general theme here with these sort of NoSQL systems, including the ones in, you know, designed by Google, is that you're kind of offloading some of this responsibility to the application—to the programmer—to sort of sort this out and make sure it's okay. Okay. And we're going to come back to that in—in just a couple of minutes.

All right, so after Bigtable, several years later, there's a paper by a bunch of folks at Google about a system called Megastore that I'm not going to spend a lot of time on, but it's basically, you know, they found this point—I sort of just made—is that these loose consistency models can complicate application programming. What they want to do is provide a little more system support for certain kinds of safe updates. Okay. So here, instead of full transactions being safe within an individual record as they are in Bigtable, they've extended it with this notion of entity groups. And so an entity group—it should be on this slide—and an entity group is a set of records that tend to go together—tend to be accessed together. Okay. So maybe again, this is the blog and all of its comments, for example. Each one of these is a record—is a sensible record store—so it's okay for them to have different schemas, but they all—they all tend to go together. Okay. And so what they've done is extend transaction support over an entire entity group—you know, a set of a set of related records. Okay. So they still get the scalability by not requiring full system-wide global, you know, synchrony, but they um allow you to sort of—they get away from this problem of, you know, I very frequently I might need to update one record and update all of its sort of children records at the same time, and I can't do that in any kind of safe way. Okay. So fine. Fast forward one more year. All right. And so there's a 2012 paper on a system called Spanner, and I just want to mention these quotes, and then we'll talk a little bit about the the system. And this one is still sort of being explored by—by the community; it's not available actually for use, but the paper is being—exploring the ideas being stored. So, for example, you don't see an open-source—actually, that's not true; you do see—there has been a couple open-source implementations of the ideas in Spanner, but they haven't—they're not quite as popular as some of the open-source implementations of the other Google systems. Okay. So, you know, it says—even though many projects happily use Bigtable, we've also consistently received complaints from users of the Bigtable—it can be difficult to use for some kinds of applications—those that have complex evolving schemas or those that want strong consistency in the presence of wide-area replication. Okay. And so then they say—go on and say—we believe it's better to have application programmers deal with performance problems due to overuse of transactions as bottlenecks arise rather than always coding around the lack of transactions. And so this, you know, the database community could have said, you know, well, sure, you know, well, duh, right? That's exactly the point is that system-supplied support for transactions is—is always a win, right? Because it's a—it's difficult, error-prone, and expensive to try to do this at the application level. And more importantly, it's fundamentally wrong in some sense to do it at the application level because it does—the application doesn't have global knowledge of what's going on, right? Only the system does. So fine. So all those—Spanner is scalable in the number of nodes—the final code here—the node-local data structures have relatively poor performance on complex SQL queries because they were designed for simple key-value accesses. And then you mentioned algorithms and data structures from the database literature can improve single-node performance a great deal. Again, you know, it's—it's somewhat of a Google style approach to the problem of—reboot everything, rebuild it all from scratch, and then sort of cherry-pick and bring things in. So this has been working pretty well, and they have fantastic impact in the community, uh, but there's a lot out there in the database literature—in the database systems—that could have been used from—from the start. I mean, in fact, trying to start from the beginning and just say we're going to build a big Google-style parallel database may have been a good choice rather than sort of getting completely away from it and then coming back incrementally and finding yourself in a SQL system now. I sort of skipped over what Spanner is; it's—it's a planet-scale database system; there is a SQL-like language—well, actually, let's go back to our—I mean, I know what I'm missing; I'm missing our—our table here; let me flip back a few. So here it is down here. So I'm—I guess I'm missing this slide here where I showed it. So really big scale primary access is—you can access by other attributes; there are transactions, and in fact, they're global this time; they're real—real asset transactions; uh, it's not clear to me whether joins are supported; I suspect they are because they keep talking about SQL, but I couldn't find an example of whether it is or not; there is some notion of schema, and they do sort of protect against data that doesn't conform to the schema; there is some notion of logical data independence, although they don't talk about it much; there is a SQL-like declarative language on top of it; I didn't see much evidence that they're doing a whole—

Lot of fancy optimization, and I did just show you that quote of where they say that their performance is sort of poor on complex analytic queries, but that's something they'll probably can could come along somewhat quickly. Okay, so fine. So that's Spanner at a high level. Let me give you a couple more details about what the system does.

So the data model here is this notion of directories, and these are a set of contiguous keys with a shared prefix. So you can think of kind of like a tablet was in Bigtable, but now they have this notion of multiple logical tables being sort of interleaved. And so if you're not used to staring at the syntax, don't worry too much, but those of you who are thinking in terms of DDL in in a relational database, they have kind of a create table language that looks like this: you create table users with two columns, and then you give it this keyword directory, and then you have a create table albums with some columns, and you have this keyword interleave in parent users. And what you end up with is something like this where there's a user with all of its albums and a user with all of its albums, as you can see here that you know what we've been talking about all these different systems are experimenting with ways of getting these nested data structures, hierarchical data structures that look a lot like what we saw way back in the 60s, right? And the motivation is the same as it was then: it's actually really, really fast when you're going to access, when you want to pull up a user and then immediately pull up all of its albums; it's really fast access this this way, right? Um, you know, but I'd probably speculate that the reasons why relational, the relational approach eventually replaced these, uh, it can and will happen here as well, is that performance is not the number one um priority a lot of time; it's it's minimizing the amount of developer headaches. So remains to be seen, but I but I think that this this incremental walk step towards a big new scalable relational database is is is underway.

Now again, that doesn't mean that I'm saying use all the old databases; they really were designed for sort of a different workload, and they really don't, there really is no evidence that they scale to some of these, some of these levels, but that doesn't mean that you sort of throw out all the, you know, everything we learned. Okay, but that's more me editorializing, so fine.

So how this works is there's a universe Master at the very, very top, and this is just a Singleton; there's only one of these for deployments, and they sort of imagine there might be only one or two of these deployments anywhere. So they have sort of one for test, one for Productions, last test and one for production, and that's it. So all, so many different applications will use the same deployment of of Spanner, and so this is mostly just status about status information about the zones; it doesn't, you know, it doesn't interact with clients at all. Then there's a placement driver that's responsible for moving these directory sets of Records around uh for load balancing purposes, and this happens on the scale of every few minutes, all right. And then within a Zone, there's a Zone master that assigns data to span servers, and there's a location proxy that sort of knows where everything is and routes requests to the appropriate span server, and then the span servers themselves serve data. And so in here, starting to look a little bit more like Bigtable, you know, zone is essentially a big to an individual Bigtable deployment. Okay, so inside of a span server, this is where the the big difference here is: this is where they're going to try to support fully consistent transactions, so across these, you know, within a group of these replicas, they can they support two-phase commit. This only is needed when a transaction actually accesses data that's that's, you know, that's across the is not not contained in one particular replica. Okay, other than that, it just skips over this this logic, and it doesn't cost anything. Okay. And then one step down below this across uh, so this is sorry, I guess I'm using the wrong terminology, so today's communities across groups, and when all when the transaction only is contained in one single group, then you drop down a level and you run the Paxos algorithm that I didn't talk about in detail, but I mentioned exists in order to sort out the region rights for uh in order to handle the right. Okay. And the only thing the other piece I'll mention here is that this term Colossus is new; it's the successor to Google File System, and Google File System is the original term for the open source implementation of HDFS which underlies MapReduce and Hadoop. Sorry, GFS is is to MapReduce as HDFS is to Hadoop. So when I'm throwing these acronyms at you, that's that's how to keep it straight. Okay.

So that's all I want to say about Spanner in particular, but just take a step back and look at all the different systems that Google has for a second. You know, MapReduce was a paper in 2004 that had a ton of impact; Bigtable had a ton of impact, and then later there's Megastore, there's the system Tenzing that we didn't talk about, but it's a SQL system on top of MapReduce, much like Hive if you were familiar with that, or if you remember we mentioned it, and then Spanner very recently. And so you can sort of organize these things into a timeline this way, just to kind of get a sense of this, and because of these systems that had so much influence, I want you to be aware of what they are and sort of how they fit together, so it doesn't just sound like a big jumble of terms. So MapReduce was, you know, the one of the earliest ones, wasn't quite the others; there's actually another one called Sawzall that didn't really get a ton of traction, but it was a nice paper, um, and then Bigtable came a couple years later and later, and I drew a dotted line there representing that they're sort of compatible or designed to go together, you know, one was MapReduce for analytics, Bigtable is for the sort of micro operations, and then both of these had a few years later had an open source implementation in Hadoop and HBase respectively. It's okay. Fast forward a few more years, and you've got Megastore and Spanner coming very quickly, one right after the other, and this heart, this heavy Blue Line represents, you know, it's pretty clear that the the influence is fairly Direct. In fact, I would suspect that there's a lot of code being borrowed; I mean, Megastore makes plenty of references to Bigtable, and Spanner makes references to both Megastore and and Bigtable, and they the papers have um many, many of the same co-authors. Okay. And then MapReduce depends directly on, sorry, excuse me, Tenzing roosters bins directly on MapReduce; it provides a SQL layer on top of MapReduce, and then there's some other systems here; one is called Dremel, which was originally for very fast aggregate queries, but really just aggregate queries, but an extremely low latency. So this is, you know, in the analytics Camp because you're doing these sort of aggregate questions as opposed to sort of micro updates, but it was extremely low latency, unlike MapReduce, which is more of a batch system, and so this is this is a a great fit, and it's a very nice system, and in fact, since then they can do joins, not just Aggregates, and more importantly, this was exposed as a service that you can just use directly over the web, even in your browser, called BigQuery, and that's a that's a that's an important one to watch; it's one of the few systems that is available as a service through, you know, as a cloud service um but scales to very, very large data and supports analytics. Okay. Okay. And then another one that we won't talk about yet, but we will come back to, is uh Pregel, and this adds the one secret ingredient that I is sort of near and dear to my heart, which is iteration, and what I mean by that is when you when you run MapReduce jobs to do analytics, you're sort of taking step one and then step two and then step three, and you stop, but for many kinds of tasks, especially in data science, we need these analytics tasks, these machine learning tasks, you have to do something again and again and again and again until some kind of convergence condition is reached, and Pregel and one of our systems and a few other systems are the ones that try to, you know, notices this limitation of MapReduce and extend it, so people were doing this with MapReduce, but they do it sort of in Fairly ad hoc ways. Okay. And so we'll come back to that and talk about it, but you know, Analytics low latency, micro updates, so there's two big classes of systems, and then analytics with iteration is perhaps a third class system that we that we'll talk about.

[Music] We can highlight those systems that were based on MapReduce itself, you know, the MapReduce paper itself in 2004, and these language layers on top, Pig and Hive in 2008, where Hive is SQL and Pig is a relational algebra looking language that we'll talk about in some detail in the next few segments, and Tenzing, which is also SQL, and Impala, which is also SQL, where Tenzing is from Google and Impala is from a company called Cloudera that's a pretty eager evangelist of MapReduce and Hadoop-based Technologies in general. Okay. So one Trend I think you see is that these declarative languages on top of the parallel processing primitive of MapReduce are really here to stay, right? So people that were relatively against these kind of languages are certainly doing it now. It's also fair to say that the uh Enterprises in general have made a pretty significant investment in SQL expertise, so even if they're attracted to the advantages that Hadoop might bring, they're they're pretty much demanding SQL. So this may be response to this uh inertia from having invested in SQL in the past; I think that's certainly true. However, it's also true that the desire for executive languages is reasonably well-founded for reasons that we've already talked about.

Okay, so you can put these systems on this timeline, and the only point I want to make about this is that there's a bit of a gap between the paper in 2004 and these systems in 2008, but then as soon as you had Hadoop, the system itself developed at Yahoo and released as an Apache open source project, you sort of immediately see an ecosystem start to emerge of extensions to it that add these, in particular adding these languages on top, and so I think that the the need for a high-level interface is motivated by how quickly they came around as soon as Hadoop was was out, and again it didn't stop with these later systems a few years later. Okay. And actually, you know, not on this page, there's potentially hundreds of, if you include, you know, research projects based on extensions to MapReduce; there are really, really a lot. Okay. And so this is some of the most popular ones, all right.

So another subset of this grid that you can look at is just the NoSQL systems. Now the whole last few segments have ostensibly been about NoSQL, but I've also included these kind of analytics systems in here, MapReduce-based systems and a few others, for example, Dremel and Spark and Shark. Dremel is a system from Google that is the back end of a query as a service system called Google BigQuery, which is pretty nice; I recommend taking a look at it. You can sort of upload data and put it in there, and it doesn't matter how big it is, and you can kind of query it at very low latency speed. Spark and Shark come from the AMP Lab at Berkeley and are part of the Berkeley data analytics stack, BDAS or BADASS, and Spark is a language label on top of it's not MapReduce, but on top of a parallel processing system, and Shark is an SQL layer even on top of that. Okay. And so a couple of the distinguishing features of Spark is it loads everything into memory, processes everything there when possible, writing things out to disk only for fault tolerance reasons, so much, much less often than MapReduce, uh, and it also supports iterative processing, which is pretty important, and we're going to come back to that a little later in the course, and then Shark again is just SQL on top of on top of this. All right. So within these NoSQL systems, one thing you can look at, there's not much I want, excuse me, there's not much I want to say about this diagram except that, you know, to point out that there's been some sort of the Cambrian explosion. So first you had sort of Memcached, which is again just this caching layer for really the real system, and the real system was a bunch of MySQL databases that weren't really working all that well for the requirements they're having used for, but you could bring things into memory and keep it there, looking it up by name, and it was just sort of a performance enhancement, a free performance enhancement if you invested in this system. The real, you know, approach of throwing out everything you have and replacing it with the NoSQL system came a little later, and so the only couple points I want to make is that there's been kind of a Cambrian explosion of different systems around this time, and that, you know, this space in here is not as in nowhere near as empty as it looks. I picked out a few systems here, but really the the emergence of new systems in the space hasn't really slowed down much at all.