目录
- 背景
- 第一部分 回本溯源
- 第二部分
HDFS大量小文件的危害 - 第三部分 小文件治理方案总结
- 第四部分 总结
- 参考文献及资料
背景
yarn.resourcemanager.store.class : 有三种StateStore，分别是基于zookeeper, HDFS, leveldb, HA高可用集群必须用ZKRMStateStore
| 存储 | yarn.resourcemanager.store.class |
|---|---|
| ZooKeeper | org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore |
| FileSystem | org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore |
| LevelDB | org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore |
By default the number of completed applications stored in state store is 10000. https://maprdocs.mapr.com/51/ReferenceGuide/Default-YARN-Parameters.html
Try to move/delete some completed applications
hadoop fs -mv /var/mapr/cluster/yarn/rm/system/FSRMStateRoot/RMAppRoot/* /path_to_local_dir
hadoop conf | grep yarn.resourcemanager.max-completed-applications
https://www.programmersought.com/article/36321434084/
https://issues.apache.org/jira/browse/YARN-7150
https://my.oschina.net/dabird/blog/3089265
https://cloud.tencent.com/developer/article/1491079
https://my.oschina.net/dabird/blog/4273830
参考文献及资料
[1] HDFS NameNode内存全景，链接：https://tech.meituan.com/2016/08/26/namenode.html