Blocks
A disk has a block size, which is the minimum amount of data that it can read or write.
Filesystems for a single disk build on this by dealing with data in blocks, which are an integral multiple of the disk block size.
Filesystem blocks are typically a few kilobytes in size, whereas disk blocks are normally 512 bytes.
This is generally transparent to the filesystem user who is simply reading or writing a file of whatever length.
However, there are tools to perform filesystem maintenance, such as df and fsck, that operate on the filesystem block level.
Namenodes and Datanodes
An HDFS cluster has two types of nodes operating in a master-worker pattern: a namenode (the master) and a number of datanodes (workers).
The namenode manages the filesystem namespace.
It maintains the filesystem tree and the metadata for all the files and directories in the tree.
This information is stored persistently on the local disk in the form of two files: the namespace image and the edit log.
The namenode also knows the datanodes on which all the blocks for a given file are located; however, it does not store block locations persistently, because this information is reconstructed from datanodes when the system starts.
Block Caching
Normally a datanode reads blocks from disk, but for frequently accessed files the blocks may be explicitly cached in the datanode’s memory, in an off-heap block cache.
By default, a block is cached in only one datanode’s memory, although the number is con- figurable on a per-file basis.
Job schedulers (for MapReduce, Spark, and other frame- works) can take advantage of cached blocks by running tasks on the datanode where a block is cached, for increased read performance.
A small lookup table used in a join is a good candidate for caching,
Federation
The namenode keeps a reference to every file and block in the filesystem in memory, which means that on very large clusters with many files, memory becomes the limiting factor for scaling .
HDFS federation, introduced in the 2.x release series, allows a cluster to scale by adding namenodes, each of which manages a portion of the filesystem namespace.
For example, one namenode might manage all the files rooted under /user, say, and a second name- node might handle files under/share.
High Availability
The combination of replicating namenode metadata on multiple filesystems and using the secondary namenode to create checkpoints protects against data loss, but it does not provide high availability of the filesystem.
The namenode is still a single point of failure (SPOF).
If it did fail, all clients-including MapReduce jobs-would be unable to read, write, or list files, because the namenode is the sole repository of the metadata and the file-to-block mapping.
In such an event, the whole Hadoop system would ef- fectively be out of service until a new namenode could be brought online.
Other courses