Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyPro

大大的周 02-07 3366

背景

作业可能会出现以下报错

ERROR org.apache.hadoop.hdfs.KeyProviderCache - Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!

这个报错是hdfs客户端的一个bug，但并不影响作业正常运行，且在2.8版本之后已经修复

相关代码

DistributedFileSystem.java 在这里调用getKeyProvider方法

public Token<?>[] addDelegationTokens( final String renewer, Credentials credentials) throws IOException { Token<?>[] tokens = super.addDelegationTokens(renewer, credentials); // 调用dfs.getKeyProvider方法 if (dfs.getKeyProvider() != null) { KeyProviderDelegationTokenExtension keyProviderDelegationTokenExtension = KeyProviderDelegationTokenExtension. createKeyProviderDelegationTokenExtension(dfs.getKeyProvider());

KeyProviderCache.java 最终会调用到createKeyProviderURI方法

private URI createKeyProviderURI(Configuration conf) { final String providerUriStr = conf.getTrimmed(DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI, ""); // 找不到dfs.encryption.key.provider.uri配置，报错 if (providerUriStr.isEmpty()) { LOG.error("Could not find uri with key [" + DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI + "] to create a keyProvider !!"); return null; } final URI providerUri; try { providerUri = new URI(providerUriStr); } catch (URISyntaxException e) { LOG.error("KeyProvider URI string is invalid [" + providerUriStr + "]!!", e.getCause()); return null; } return providerUri; } 解决

社区已有相关patch https://issues.apache.org/jira/browse/HDFS-7931

代码比较简单，把判断条件换成了检查配置是否存在 DFSClient.java

+ public boolean isHDFSEncryptionEnabled() { + return conf.get( + DFSConfigKeys.DFS_ENCRYPTION_KEY_PROVIDER_URI, null) != null; + }

DistributedFileSystem.java

public Token<?>[] addDelegationTokens( final String renewer, Credentials credentials) throws IOException { Token<?>[] tokens = super.addDelegationTokens(renewer, credentials); - if (dfs.getKeyProvider() != null) { + if (dfs.isHDFSEncryptionEnabled()) { KeyProviderDelegationTokenExtension keyProviderDelegationTokenExtension = KeyProviderDelegationTokenExtension. createKeyProviderDelegationTokenExtension(dfs.getKeyProvider());