Log-structured file system

Transcription

That brings me to another background technology that I have to explain to you and which is called log structured file system. The idea here is that when I make a change to file y meaning I either append to the file or make some modifications to it. What I'm going to do is rather than writing the file as is, I'm going to write the change that I made to the file as a log record. So, I have a log record that says, what are the changes I made to this file x. Similarly, I have a log record of all the changes I made to this file y. And this is being done in a data structure which I'll call log segment. And I'll keep this log segment data structure in memory, of course, to make it fast in terms of the file system operation. So with this log segment data structure, what I can do is, buffer the changes to multiple files in one contiguous log segment data structure. So this log segment data structure, I can write it out as a file, and when I write it out, I'm not writing a single file, but I'm actually writing a log segment which contains all the changes made to multiple files. And because the log segment is contiguous, I can write it sequentially on the disk and sequential writes are good in the disk subsystem. And what we want to do is, we want to gather these changes to files that are happening in my system in the log segment in memory, and every once in a while, flush the log segment to disk, once the log segment fills up to a certain extent, or periodically. And the reason, of course, is the fact that if it is in memory, you have to worry about reliability of your file system, if, in fact, the node crashes. And therefore, what we want to do is, we want to either write out these log segments periodically or when a lot of file activity is happening and the log segment fills up very rapidly. After it passes of threshold, then you write it out to the desk. So in other words, we use a space metric or a time metric to figure out when to flush the changes from the log segment into the disk. And this solves the small write problem because if y happens to be a small file. No problem, because we are not writing y as-is on to the disk. But what we are writing is this log segment that contains changes that have been made to y in addition to changes that have been made to a number of other files. And therefore, this log segment is going to be a big file. And therefore, we can use the RAID technology to stripe the log segments across multiple disks. And give the benefit of the parallel IO that's possible with the RAID technology. So this log structured file system solves the small write problem. And in log structured file system, there are only logs. No data files. You'll never write any data files. All the things that you're writing are these append only logs to the disk. And when you have a read of a file, the read of a file, if it has to go to the disk and fetch that file, then the file system has to reconstruct the file from the logs that it has stored on the disk. Of course, once it comes into the memory of the server, then in the file cache the file is going to remain as a file. But, if at any point, the server has to fetch the file from the disk, it's actually fetching the log segments. And then reconstructing the file from the log segments. That's important. Which means that, in a log structured file system, there could be latency associated with reading a file for the first time from the disk. Of course, once it is read from the disk and reconstructed, it is in memory. In the file cache of the server, then everything is fine. But the first time, you have to read it from the disk, it's going to take some time because you have to read all these log segments and reconstruct it and that's where parallel RAID technology can be very helpful, because you're aggregating all the bandwidth that's available for reading the log segments from multiple disks at the same time. And the other thing that you have to worry about, when you have a log structured file system, is that these logs represent changes that have been made to the files. So, for instance, I may have written a particular block of y and that may be the change sitting here. Next time, what I'm doing is perhaps I'm writing the same block of the file. In which case, the first strike that I did, that is invalid. I have got a new write of that same block. So, you see that over time, the logs are going to have lots of holes created by overwriting the same block of a particular file. So in a log structured file system, one of the things that has to happen is that the logs have to be cleaned periodically to ensure that the disk is not cluttered with wasted logs that have empty holes in them because of old writes to parts of a file that are no longer relevant. Because those parts of the file have been rewritten, overwritten by subsequent writes to the same file. So logs, as I've introduced you, is similar to the disks that you've seen in the DSM system with the multiple writer protocol that we talked about in a previous lecture. You may have also heard the term, journalling file system, there is a difference between log structured file system, and journalling file system. Journalling file systems has both log files as well as data files, and what a journalling file system does, is it applies the log files to the data files and discards the log files. The goal is similar in a journaling file system, and the goal is to solve the small write problem, but in a journaling file system, the logs are there only for a short duration of time before the logs are committed to the data files themselves. Whereas in a log structured file system, you don't have data files at all, all that you have are log files and reads have to deconstruct the data from the log files.