扫描并从依赖关系jar中读取文件

Question

我正在开发一个使用大型数据集的spring引导命令行应用程序。数据集很麻烦，所以我想将它与我的代码分开，但我不想让用户不得不管理数据。

我认为最理想的解决方案是创建一个数据jar并从中读取主应用程序。不幸的是，当它在一个依赖关系jar中时，我无法成功读取数据。

我做了一个示例应用程序来证明什么，我至今试图

https://github.com/LewisWatson/java-data-jar

. ├── data-jar │   ├── pom.xml │   └── src │   └── main │   └── resources │   └── data │   ├── hellos │   │   └── hello.txt │   └── loremIpsums │   └── loremIpsum.txt ├── data-jar-reader │   ├── pom.xml │   └── src │   └── main │   ├── java │   │   └── com │   │   └── example │   │   └── datajarreader │   │   └── DataJarReaderApplication.java │   └── resources │   └── application.properties ├── LICENSE ├── pom.xml └── README.md

它有两个模块：

• data-jar包含两个文本文件在maven的标准资源目录中的一个data目录中有两个目录。
• data-jar-reader是弹簧引导命令行应用程序，它依赖于data-jar并尝试访问其数据文件。

，做阅读类是DataJarReaderApplication

@SpringBootApplication 
public class DataJarReaderApplication implements CommandLineRunner { 

    private static final Logger log = LoggerFactory.getLogger(DataJarReaderApplication.class); 

    FileSystem fileSystem; 

    public static void main(String[] args) { 
    SpringApplication.run(DataJarReaderApplication.class, args); 
    } 

    @Override 
    public void run(String... args) throws Exception { 
    Path path = getTestDataPath(); 
    List<String> data = loadTestData(path); 
    log.info("data {}", data); 
    } 

    private Path getTestDataPath() throws URISyntaxException, IOException { 

    Path path; 

    URI uri = this.getClass().getResource("/data").toURI(); 
    log.info("uri: {}", uri); 

    if (uri.getScheme().equals("jar")) { 
     fileSystem = FileSystems.newFileSystem(uri, Collections.<String, Object>emptyMap()); 
     log.info("fileSystem: {}", fileSystem); 
     path = fileSystem.getPath("/data"); 
    } else { // not in a jar, probably running in iDE 
     path = Paths.get(uri); 
    } 

    return path; 
    } 

    private List<String> loadTestData(Path testDataDirectory) throws IOException { 
    try (Stream<Path> path = Files.walk(testDataDirectory).sorted()) { 
     return loadTrackData(path); 
    } 
    } 

    private List<String> loadTrackData(Stream<Path> walk) throws IOException { 

    List<String> data = new ArrayList<>(); 

    for (Iterator<Path> it = walk.iterator(); it.hasNext();) { 

     Path path = it.next(); 

     if (path.getFileName().toString().endsWith(".txt")) { 

     log.info("text file path: {}", path); 

     List<String> dataFromFile = getData(path); 
     data.addAll(dataFromFile); 

     } else { 

     log.info("non text file path: {}", path); 

     } 

    } 

    return data; 
    } 

    private List<String> getData(Path path) throws IOException { 

    List<String> data = new ArrayList<>(); 

    try (BufferedReader reader = Files.newBufferedReader(path, StandardCharsets.UTF_8)) { 

     String line; 

     while ((line = reader.readLine()) != null) { 
     data.add(line); 
     } 

    } 
    return data; 
    } 
} 

输出我得到的，当我运行的应用表明，它的查找数据在数据罐子

jar:file:/home/lewis/workspace/java-data-jar/data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/data-jar-1.0-SNAPSHOT.jar!/data 

但是当它试图访问数据我得到一个没有这样的文件例外...

$ java -jar data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar 

    . ____   _   __ _ _ 
/\\/___'_ __ _ _(_)_ __ __ _ \ \ \ \ 
(()\___ | '_ | '_| | '_ \/ _` | \ \ \ \ 
\\/ ___)| |_)| | | | | || (_| | )))) 
    ' |____| .__|_| |_|_| |_\__, |//// 
=========|_|==============|___/=/_/_/_/ 
:: Spring Boot ::  (v1.5.7.RELEASE) 

2017-10-16 09:35:46.238 INFO 18756 --- [   main] c.e.d.DataJarReaderApplication   : Starting DataJarReaderApplication v0.0.1-SNAPSHOT on mir with PID 18756 (/home/lewis/workspace/java-data-jar/data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar started by lewis in /home/lewis/workspace/java-data-jar) 
2017-10-16 09:35:46.249 INFO 18756 --- [   main] c.e.d.DataJarReaderApplication   : No active profile set, falling back to default profiles: default 
2017-10-16 09:35:46.376 INFO 18756 --- [   main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@443b7951: startup date [Mon Oct 16 09:35:46 BST 2017]; root of context hierarchy 
2017-10-16 09:35:47.049 INFO 18756 --- [   main] o.s.j.e.a.AnnotationMBeanExporter  : Registering beans for JMX exposure on startup 
2017-10-16 09:35:47.056 INFO 18756 --- [   main] c.e.d.DataJarReaderApplication   : uri: jar:file:/home/lewis/workspace/java-data-jar/data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/data-jar-1.0-SNAPSHOT.jar!/data 
2017-10-16 09:35:47.094 INFO 18756 --- [   main] c.e.d.DataJarReaderApplication   : fileSystem: /home/lewis/workspace/java-data-jar/data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar 
2017-10-16 09:35:47.098 INFO 18756 --- [   main] utoConfigurationReportLoggingInitializer : 

Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled. 
2017-10-16 09:35:47.111 ERROR 18756 --- [   main] o.s.boot.SpringApplication    : Application startup failed 

java.lang.IllegalStateException: Failed to execute CommandLineRunner 
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:735) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE] 
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:716) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE] 
    at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:703) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE] 
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:304) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE] 
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE] 
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE] 
    at com.example.datajarreader.DataJarReaderApplication.main(DataJarReaderApplication.java:34) [classes!/:0.0.1-SNAPSHOT] 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131] 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131] 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131] 
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131] 
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [data-jar-reader-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT] 
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [data-jar-reader-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT] 
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [data-jar-reader-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT] 
    at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [data-jar-reader-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT] 
Caused by: java.nio.file.NoSuchFileException: /data 
    at com.sun.nio.zipfs.ZipPath.getAttributes(ZipPath.java:666) ~[zipfs.jar:1.8.0_131] 
    at com.sun.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:294) ~[zipfs.jar:1.8.0_131] 
    at java.nio.file.Files.readAttributes(Files.java:1737) ~[na:1.8.0_131] 
    at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219) ~[na:1.8.0_131] 
    at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) ~[na:1.8.0_131] 
    at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322) ~[na:1.8.0_131] 
    at java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:72) ~[na:1.8.0_131] 
    at java.nio.file.Files.walk(Files.java:3574) ~[na:1.8.0_131] 
    at java.nio.file.Files.walk(Files.java:3625) ~[na:1.8.0_131] 
    at com.example.datajarreader.DataJarReaderApplication.loadTestData(DataJarReaderApplication.java:63) [classes!/:0.0.1-SNAPSHOT] 
    at com.example.datajarreader.DataJarReaderApplication.run(DataJarReaderApplication.java:40) [classes!/:0.0.1-SNAPSHOT] 
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:732) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE] 
    ... 14 common frames omitted 

Answer 1

如果您在IDE中运行应用程序，则不必以不同方式访问数据。这些数据文件总是资源，例如文件在类路径中，应该作为资源加载。

Spring有一个方便的资源加载器，可以帮助发现类路径中的数据文件：PathMatchingResourcePatternResolver。

例如，让所有.txt文件中data包：

ClassLoader cl = this.getClass().getClassLoader(); 
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(cl); 
Resource[] resources = resolver.getResources("classpath:/data/**/*.txt") ; 
for (Resource resource: resources){ 
    logger.info(resource.getFilename()); 
} 

然后可以使用getInputStream每个资源阅读您的文件。

Answer 2

假设你真的想使用该JAR，你有几种选择：

• 可以列出类路径的数据罐子和使用ClassLoader.getResourceAsStream（relativePathWithinJar）。
• 您可以将数据jar放入某个目录，并使用Java API将其作为常规ZIP/JAR归档打开。

在第二种情况下，看看JarFile类。要打开它，你必须提供jar文件的路径。您可以把它变成硬编码目录，到当前目录或通过传递命令行参数，系统属性或类似的东西，一些任意路径。打开它之后，可以通过它们的相对（jar）路径将各个文件作为JarEntries获取。

与任何大型文件，最好避免使用整个数据加载到内存中的方法和坚持，对于连续的，按需提供阅读的InputStream方法。