hadoop - multiple Datanode directories write operation -


iam using hadoop apache 2.7.1 , have configured data node directory have multiple directories

        <property>                  <name>dfs.data.dir</name>                  <value>/opt/hadoop/data_dir,file:///hdd/data_dir/</value>                  <final>true</final>         </property> 

according configuration writing file data should happens on both directories /opt/hadoop/data_dir , file:///hdd/data_dir/ same blocks names , on same sub directories names

but in cluster behavior not happening times writes data blocks local directory /opt/hadoop/data_dir , times writes data blocks

external hard directory file:///hdd/data_dir

what possible reasons , how control behavior

your interpretation of dfs.data.dir wrong. stated:

with same blocks names , on same sub directories names

files might replicated happens on different datanodes, not on same datanode different directories. property dfs.data.dir deprecated , should dfs.datanode.data.dir has been there since hadoop 2.

from hdfs-default.xml

dfs.datanode.data.dir

determines on local filesystem dfs data node should store blocks. if comma-delimited list of directories, data stored in named directories, typically on different devices.


Comments

Popular posts from this blog

android - InAppBilling registering BroadcastReceiver in AndroidManifest -

python Tkinter Capturing keyboard events save as one single string -

sql server - Why does Linq-to-SQL add unnecessary COUNT()? -