lein uberjar does not handle signed dependency jars correctly #31

ogrisel · 2010-03-26T13:31:39Z

If one of the dependencies is signed, creating a standalone jar with "lein uberjar" yields the following exception when executing the main class of the project:

Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes

In my case the inclusion of a "META-INF/DUMMY.SF" file from one of the depedency (janino-2.5.15.jar) is the root of the problem. Patching leiningen as follows in the attahed patch fixes the problem (however this is probably not the right way to fix it).

The text was updated successfully, but these errors were encountered:

ogrisel · 2010-03-26T13:32:34Z

diff --git a/src/leiningen/uberjar.clj b/src/leiningen/uberjar.clj
index 67ebcc1..2e9b2cd 100644
--- a/src/leiningen/uberjar.clj
+++ b/src/leiningen/uberjar.clj
@@ -53,7 +53,7 @@ may wish to clean first."
                     (filter #(.endsWith (.getName %) ".jar"))
                     (cons (file (:root project) (str (:name project) ".jar"))))
           [_ components] (reduce (partial include-dep out)
-                                 [#{"META-INF/plexus/components.xml"} nil]
+                                 [#{"META-INF/plexus/components.xml" "META-INF/DUMMY.SF"} nil]
                                  deps)]
       (when-not (empty? components)
         (.putNextEntry out (ZipEntry. "META-INF/plexus/components.xml"))

michalmarczyk · 2010-06-25T22:47:57Z

I'm planning to look into this soonish, though I won't promise to be particularly fast (since I'm not entirely sure what's involved). Should anyone plan on beating me to it, please let me know. :-)

ogrisel · 2010-06-25T22:49:56Z

In the mean time it's good to have following command at hand:
zip *-standalone.jar -d META-INF/DUMMY.SF

michalmarczyk · 2010-06-25T23:01:17Z

Thanks for the tip!

technomancy · 2010-11-13T19:28:49Z

Is removing META-INF/DUMMY.SF the correct solution for all signed jars, or just for janino?

ogrisel · 2010-11-13T20:01:41Z

No idea. It was the first and only time I encountered the issue and found this file by binary search with trial and error (removing half remaining files at a time).

technomancy · 2010-11-13T23:05:44Z

According to an older thread on the mailing list sometimes it has another name. If there's a regex that can always catch it then we can make :jar-exclusions default to that regex.

fdhenard · 2011-06-29T20:57:48Z

This issue occurred for me as well with the SQL Server jdbc driver. I used @ogrisel's zip *-standalone.jar -d META-INF/DUMMY.SF and it worked for me.

It would be nice if this issue was re-opened and fixed

technomancy · 2011-06-29T21:32:35Z

Is it safe to always exclude META-INF/DUMMY.SF from uberjars? I can do this (in a way that's easy to override) if it's appropriate.

fdhenard · 2011-06-29T22:01:44Z

I don't know about that. I actually had to remove zigbert.sf from the standalone.jar, so I believe it's specific to which .jars have the security signature file. So I ran zip *-standalone.jar -d META-INF/zigbert.sf

technomancy · 2011-06-29T22:45:37Z

It would be nice if this issue was re-opened and fixed

So do you have a solution to propose? Right now it's already easy enough to just add :uberjar-exclusions [#"zigbert.sf"] to project.clj as the need arises. I don't see a general-purpose fix being possible if the name of the file at fault varies.

fdhenard · 2011-06-30T14:33:34Z

I'm very sorry that I missed that in the conversation above. I just tried it and it worked great. Thanks!

timmc · 2012-01-06T01:56:51Z

[gnu.getopt/java-getopt "1.0.13"] has the same problem, but with TESTKEY.SF. I think any SF is going to be a problem.

timmc · 2012-01-09T00:37:49Z

Failing testcase:

project.clj

(defproject clj "1.0.0-SNAPSHOT"
  :description "FIXME: write description"
  :dependencies [[org.clojure/clojure "1.3.0"]
                 [gnu.getopt/java-getopt "1.0.13"]] ;; jar contains META-INF/TESTKEY.SF
  :main clj.core)

src/clj/core.clj

(ns clj.core
  (:gen-class))

(defn -main [& args]
  (println "hi"))

Explanation

Jars may be signed with multiple signature files, which take the form of #"^META-INF/[a-zA-Z0-9_-]*.SF$" and corresponding .DSA and .RSA key files. The main problem is that the signature file appears to sign the MANIFEST.MF, which Leiningen overwrites:

SHA1-Digest-Manifest-Main-Attributes: j6FYdHNwvIzMP83Z6mTnxQZekCo=
SHA1-Digest-Manifest: WBCxJyIwje6BE6eYCdhrnB0c7Yk=

http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/jarsigner.html

Patch

Forthcoming as pull request.

… meta-inf. Also make case-insensitive and anchor the end.

sneilan · 2012-01-12T21:31:56Z

I'm having problems with this too.

timmc · 2012-01-13T00:23:43Z

sneilan: I've submitted a patch, but until a fix is released, I'm using this in my project.clj:

:uberjar-exclusions [#"(?i)^META-INF/[^/]*\.SF$"]

Patch for reopened #31 (third time's the charm)

technomancy · 2012-01-25T18:19:18Z

Looks like this is taken care of.

sneilan · 2012-01-27T20:14:08Z

It works!! Thank you!

sneilan · 2012-01-28T00:36:56Z

I found one more issue.

If I include mmemail as a dependency and run the standalone jar:

Exception in thread "main" java.lang.SecurityException: no manifiest section for signature file entry javax/activation/MimeType.class
at sun.security.util.SignatureFileVerifier.verifySection(SignatureFileVerifier.java:392)
at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:249)
at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:193)
at java.util.jar.JarVerifier.processEntry(JarVerifier.java:251)
at java.util.jar.JarVerifier.update(JarVerifier.java:205)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:338)
at java.util.jar.JarFile.getInputStream(JarFile.java:403)
at sun.misc.JarIndex.getJarIndex(JarIndex.java:116)
at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:623)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:614)
at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:598)
at sun.misc.URLClassPath$3.run(URLClassPath.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:337)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:314)
at sun.misc.URLClassPath.getResource(URLClassPath.java:184)
at java.net.URLClassLoader$1.run(URLClassLoader.java:209)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: gamespot.core. Program will exit.

This is because the lines that look like
Name: javax/activation/DataHandler.class
SHA1-Digest: 6X6xyd3/vc4l1Rm9JKwlSR1eFwI=
from /META-INF/MANIFEST.MF in the mmemail jar aren't included in the standalone jar's /META-INF/MANIFEST.MF.

I add those lines in and it works just fine.

This is with leiningen 1.6.2
sneilan@sneilan-ubuntu:~/BucketsOfNantucket/research/gamespot$ lein -v
Leiningen 1.6.2 on Java 1.6.0_23 OpenJDK 64-Bit Server VM

sneilan · 2012-01-28T00:42:42Z

Or, if I get rid of the SHA1-Digest lines from the MANIFEST.MF in the mmemail jar, it also works! (I also got rid of the two SUN....RSA whatever files.)

This makes everything better again :)

technomancy · 2012-01-28T04:05:10Z

I'm not sure what this means. Is there more that uberjar should be doing?

ifesdjeen · 2012-05-12T14:41:30Z

This issue still exists in Lein2, haven't tried yet with Master.
But whenever I include Jetty, I keep getting these:

Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
        at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:221)
        at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:176)
        at java.util.jar.JarVerifier.processEntry(JarVerifier.java:288)
        at java.util.jar.JarVerifier.update(JarVerifier.java:199)
        at java.util.jar.JarFile.initializeVerifier(JarFile.java:323)
        at java.util.jar.JarFile.getInputStream(JarFile.java:388)
        at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:692)
        at sun.misc.Resource.cachedInputStream(Resource.java:61)
        at sun.misc.Resource.getByteBuffer(Resource.java:144)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:256)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

michaelklishin · 2012-05-12T16:02:05Z

With what lein2 version? Lein2 Preview 4 was released less than 24 hours ago.

ifesdjeen · 2012-05-12T17:27:29Z

Checked on preview4 just now. Issue is reproducible.

technomancy · 2012-05-12T21:06:09Z

I just made an uberjar containing Jetty with preview4, so if this is still happening I will need some more details to reproduce the problem.

ifesdjeen · 2012-05-12T21:57:04Z

I will check out, maybe it was some other dependency. Will come back with small repro repo or good news.

alex p

Am 12.05.2012 um 23:06 schrieb Phil [email protected]:

I just made an uberjar containing Jetty with preview4, so if this is still happening I will need some more details to reproduce the problem.

Reply to this email directly or view it on GitHub:
#31 (comment)

ifesdjeen · 2012-05-13T10:47:12Z

Actually, it was jetty-server.

Here's the repository where the bug reproduces: https://github.com/ifesdjeen/lein-manifest-bug-reproduce/blob/master/project.clj#L4

It may be actually not a Leningen bug, I understand that. If you could give me a hint what that could be, i'll dig & talk to Ring guys, maybe, as they're using it https://github.com/mmcgrana/ring/blob/master/project.clj#L7

On the side note, I tried also the forementioned java-getopt, and issue reproduced:

[gnu.getopt/java-getopt "1.0.13"]

That also causes the issue:

 [org.eclipse.jetty/jetty-server "7.6.1.v20120215"]

Proof I have correct lein :)

lein2 version
Leiningen 2.0.0-preview4 on Java 1.6.0_31 Java HotSpot(TM) 64-Bit Server VM

And the command I run:

lein2 uberjar && java -Xmx1024m -jar target/bugreproduce-1.0.0-SNAPSHOT-standalone.jar

And output:

All namespaces already :aot compiled.
Created /Users/alexp/p/bugreproduce/target/bugreproduce-1.0.0-SNAPSHOT.jar
Including bugreproduce-1.0.0-SNAPSHOT.jar
Including jetty-util-7.6.1.v20120215.jar
Including jetty-io-7.6.1.v20120215.jar
Including jetty-http-7.6.1.v20120215.jar
Including jetty-continuation-7.6.1.v20120215.jar
Including javax.servlet-2.5.0.v201103041518.jar
Including jetty-server-7.6.1.v20120215.jar
Including clojure-1.3.0.jar
Created /Users/alexp/p/bugreproduce/target/bugreproduce-1.0.0-SNAPSHOT-standalone.jar
Exception in thread "main" java.lang.SecurityException: Invalid signature file digest for Manifest main attributes
    at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:221)
    at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:176)
    at java.util.jar.JarVerifier.processEntry(JarVerifier.java:288)
    at java.util.jar.JarVerifier.update(JarVerifier.java:199)
    at java.util.jar.JarFile.initializeVerifier(JarFile.java:323)
    at java.util.jar.JarFile.getInputStream(JarFile.java:388)
    at sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:692)
    at sun.misc.Resource.cachedInputStream(Resource.java:61)
    at sun.misc.Resource.getByteBuffer(Resource.java:144)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:256)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Thank you!

technomancy · 2012-05-21T20:29:26Z

Looks like the regex here just got goofed in 2.x.

…n uberjars.

ghoseb · 2012-05-30T11:01:07Z

This problem occurs with Jetty (via ring) as well, because Jetty recently moved to the Eclipse Foundation and they are now signing the JAR files.

Here is a minimal test-case to reproduce -

;; project.clj
(defproject leintest "0.1.0-SNAPSHOT"
  :description "FIXME: write description"
  ;; this fixes the problem
  ;; :uberjar-exclusions [#"(?i)^META-INF/[^/]*\.(SF|RSA)$"]
  :dependencies [[org.clojure/clojure "1.4.0"]
                 ;; bringing in [ring/ring-jetty-adapter "1.1.0"] works as well
                 [org.eclipse.jetty/jetty-server "8.1.4.v20120524"]]
  :main leintest.core)

;; src/leintest/core.clj
(ns leintest.core
  (:gen-class)
  (:import org.eclipse.jetty.server.Server))

(defn -main
  [& args]
  (println "Hello, world!"))

I have issued a pull request that augments the previous regexp to exclude .RSA files as well. We might need to keep changing the regexp in the future as and when we discover more file-types.

ghost · 2014-10-08T12:55:32Z

Here is my solution!
http://middlesphere-1.blogspot.ru/2014/06/how-to-make-jar-with-dependencies-using.html

The main idea is to put all dependecies into ./lib folder

technomancy reopened this Jan 8, 2012

timmc added a commit to timmc/leiningen that referenced this issue Jan 9, 2012

technomancy#31: Expand JAR signature file matcher to all .sf files in…

032e892

… meta-inf. Also make case-insensitive and anchor the end.

technomancy added a commit that referenced this issue Jan 13, 2012

Merge pull request #371 from timmc/31-sf-star

97f5fb7

Patch for reopened #31 (third time's the charm)

technomancy closed this as completed Jan 25, 2012

technomancy mentioned this issue May 16, 2012

Invalid signature file digest for Manifest main attributes (lein2) #587

Closed

technomancy reopened this May 18, 2012

technomancy closed this as completed in 75c2508 May 21, 2012

ghoseb added a commit to ghoseb/leiningen that referenced this issue May 30, 2012

Fix technomancy#31: Exclude RSA signature files from being included i…

909783f

…n uberjars.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lein uberjar does not handle signed dependency jars correctly #31

lein uberjar does not handle signed dependency jars correctly #31

ogrisel commented Mar 26, 2010

ogrisel commented Mar 26, 2010

michalmarczyk commented Jun 25, 2010

ogrisel commented Jun 25, 2010

michalmarczyk commented Jun 25, 2010

technomancy commented Nov 13, 2010

ogrisel commented Nov 13, 2010

technomancy commented Nov 13, 2010

fdhenard commented Jun 29, 2011

technomancy commented Jun 29, 2011

fdhenard commented Jun 29, 2011

technomancy commented Jun 29, 2011

fdhenard commented Jun 30, 2011

timmc commented Jan 6, 2012

timmc commented Jan 9, 2012

sneilan commented Jan 12, 2012

timmc commented Jan 13, 2012

technomancy commented Jan 25, 2012

sneilan commented Jan 27, 2012

sneilan commented Jan 28, 2012

sneilan commented Jan 28, 2012

technomancy commented Jan 28, 2012

ifesdjeen commented May 12, 2012

michaelklishin commented May 12, 2012

ifesdjeen commented May 12, 2012

technomancy commented May 12, 2012

ifesdjeen commented May 12, 2012

ifesdjeen commented May 13, 2012

technomancy commented May 21, 2012

ghoseb commented May 30, 2012

ghost commented Oct 8, 2014

lein uberjar does not handle signed dependency jars correctly #31

lein uberjar does not handle signed dependency jars correctly #31

Comments

ogrisel commented Mar 26, 2010

ogrisel commented Mar 26, 2010

michalmarczyk commented Jun 25, 2010

ogrisel commented Jun 25, 2010

michalmarczyk commented Jun 25, 2010

technomancy commented Nov 13, 2010

ogrisel commented Nov 13, 2010

technomancy commented Nov 13, 2010

fdhenard commented Jun 29, 2011

technomancy commented Jun 29, 2011

fdhenard commented Jun 29, 2011

technomancy commented Jun 29, 2011

fdhenard commented Jun 30, 2011

timmc commented Jan 6, 2012

timmc commented Jan 9, 2012

Failing testcase:

project.clj

src/clj/core.clj

Explanation

Patch

sneilan commented Jan 12, 2012

timmc commented Jan 13, 2012

technomancy commented Jan 25, 2012

sneilan commented Jan 27, 2012

sneilan commented Jan 28, 2012

sneilan commented Jan 28, 2012

technomancy commented Jan 28, 2012

ifesdjeen commented May 12, 2012

michaelklishin commented May 12, 2012

ifesdjeen commented May 12, 2012

technomancy commented May 12, 2012

ifesdjeen commented May 12, 2012

ifesdjeen commented May 13, 2012

technomancy commented May 21, 2012

ghoseb commented May 30, 2012

ghost commented Oct 8, 2014