(For more resources related to this topic, see here.)
In the early days of Java, when projects were small and didn’t have many external dependencies, developers tended to manage dependencies manually by copying the required JAR files in the lib folder and checking it in their SCM/VCS with their code. This is still followed by a lot of developers, even today. But due to the aforementioned issues, this is not an option for larger projects.
In many enterprises, there are central servers, FTP, shared drives, and so on, which store the approved libraries for use and also internally released libraries. But managing and tracking them manually is never easy. They end up relying on scripts and build files.
Maven came and standardized this process. Maven defines standards for the project format to define its dependencies, formats for repositories to store libraries, the automated process to fetch transitive dependencies, and much more.
Most of the systems today either back onto Maven’s dependency management system or on Ivy’s, which can function in the same way, and also provides its own standards, which is heavily inspired by Maven. SBT uses Ivy in the backend for dependency management, but uses a custom DSL to specify the dependency.
Quick introduction to Maven or Ivy dependency management
Apache Maven is not a dependency management tool. It is a project management and a comprehension tool. Maven is configured using a Project Object Model (POM), which is represented in an XML file. A POM has all the details related to the project right from the basic ones, such as groupId, artifactId, version, and so on, to environment settings such as prerequisites, and repositories.
Apache Ivy is a dependency management tool and a subproject of Apache Ant. Ivy integrates publicly available artifact repositories automatically. The project dependencies are declared using XML in a file called ivy.xml. This is commonly known as the Ivy file.
Ivy is configured using a settings file. The settings file (ivysettings.xml) defines a set of dependency resolvers. Each resolver points to an Ivy file and/or artifacts. So, the configuration essentially indicates which resource should be used to resolve a module.
How Ivy works
The following diagram depicts the usual cycle of Ivy modules between different locations:
The tags along the arrows are the Ivy commands that need to be run for that task, which are explained in detail in the following sections.
Resolve
Resolve is the phase where Ivy resolves the dependencies of a module by accessing the Ivy file defined for that module.
For each dependency in the Ivy file, Ivy finds the module using the configuration. A module could be an Ivy file or artifact. Once a module is found, its Ivy file is downloaded to the Ivy cache. Then, Ivy checks for the dependencies of that module. If the module has dependencies on other modules, Ivy recursively traverses the graph of dependencies, handling conflicts simultaneously.
After traversing the whole graph, Ivy downloads all the dependencies that are not already in the cache and have not been evicted by conflict management. Ivy uses a filesystem-based cache to avoid loading dependencies already available in the cache.
In the end, an XML report of the dependencies of the module is generated in the cache.
Retrieve
Retrieve is the act of copying artifacts from the cache to another directory structure. The destination for the files to be copied is specified using a pattern. Before copying, Ivy checks if the files are not already copied to maximize performance. After dependencies have been copied, the build becomes independent of Ivy.
Publish
Ivy can then be used to publish the module to a repository. This can be done by manually running a task or from a continuous integration server.
Dependency management in SBT
In SBT, library dependencies can be managed in the following two ways:
- By specifying the libraries in the build definition
- By manually adding the JAR files of the library
Manual addition of JAR files may seem simple in the beginning of a project. But as the project grows, it may depend on a lot of other projects, or the projects it depends on may have newer versions. These situations make handling dependencies manually a cumbersome task. Hence, most developers prefer to automate dependency management.
Automatic dependency management
SBT uses Apache Ivy to handle automatic dependency management. When dependencies are configured in this manner, SBT handles the retrieval and update of the dependencies. An update does not happen every time there is a change, since that slows down all the processes. To update the dependencies, you need to execute the update task. Other tasks depend on the output generated through the update. Whenever dependencies are modified, an update should be run for these changes to get reflected. There are three ways in which project dependencies can be specified. They are as follows:
- Declarations within the build definition
- Maven dependency files, that is, POM files
- Configuration and settings files used for Ivy
- Adding JAR files manually
Declaring dependencies in the build definition
The Setting key libraryDependencies is used to configure the dependencies of a project. The following are some of the possible syntaxes for libraryDependencies:
- libraryDependencies += groupID % artifactID % revision
- libraryDependencies += groupID %% artifactID % revision
- libraryDependencies += groupID % artifactID % revision % configuration
- libraryDependencies ++= Seq(
groupID %% artifactID % revision,
groupID %% otherID % otherRevision
)
Let’s explain some of these examples in more detail:
- groupID: This is the organization/group’s ID by whom it was published
- artifactID: This is the project’s name on which there is a dependency
- revision: This is the Ivy revision of the project on which there is a dependency
- configuration: This is the Ivy configuration for which we want to specify the dependency
Notice that the first and second syntax are not the same. The second one has a %% symbol after groupID. This tells SBT to append the project’s Scala version to artifactID.
So, in a project with Scala Version 2.9.1, libraryDependencies ++= Seq(“mysql” %% “mysql-connector-java” % “5.1.18”) is equivalent to libraryDependencies ++= Seq(“mysql” % “mysql-connector-java_2.9.1” % “5.1.18”).
The %% symbol is very helpful for cross-building a project. Cross-building is the process of building a project for multiple Scala versions. SBT uses the crossScalaVersion key’s value to configure dependencies for multiple versions of Scala. Cross-building is possible only for Scala Version 2.8.0 or higher.
The %% symbol simply appends the current Scala version, so it should not be used when you know that there is no dependency for a given Scala version, although it is compatible with an older version. In such cases, you have to hardcode the version using the first syntax.
Using the third syntax, we could add a dependency only for a specific configuration. This is very useful as some dependencies are not required by all configurations. For example, the dependency on a testing library is only for the test configuration. We could declare this as follows:
libraryDependencies ++= Seq("org.specs2" % "specs2_2.9.1" % "1.12.3" % "test")
We could also specify dependency for the provided scope (where the JDK or container provides the dependency at runtime).This scope is only available on compilation and test classpath, and is not transitive. Generally, servlet-api dependencies are declared in this scope:
libraryDependencies += "javax.servlet" % "javax.servlet-api" % "3.0.1" % "provided"
The revision does not have to be a single-fixed version, that is, it can be set with some constraints, and Ivy will select the one that matches best. For example, it could be latest integration or 12.0 or higher, or even a range of versions.
A URL for the dependency JAR
If the dependency is not published to a repository, you can also specify a direct URL to the JAR file:
libraryDependencies += groupID %% artifactID % revision from directURL
directURL is used only if the dependency cannot be found in the specified repositories and is not included in published metadata. For example:
libraryDependencies += "slinky" % "slinky" % "2.1" from "http://slinky2.googlecode.com/svn/artifacts/2.1/slinky.jar"
Extra attributes
SBT also supports Ivy’s extra attributes. To specify extra attributes, one could use the extra method. Consider that the project has a dependency on the following Ivy module:
<ivy-module version ="2.0" > <info organization="packt" module = "introduction" e:media = "screen" status = "integration" e:codeWord = "PP1872"</ivy-module>
A dependency on this can be declared by using the following:
libraryDependencies += "packt" % "introduction" % "latest.integration" extra( "media"->"screen", "codeWord"-> "PP1872")
The extra method can also be used to specify extra attributes for the current project, so that when it is published to the repository its Ivy file will also have extra attributes. An example for this is as follows:
projectID << projectID {id => id extra( "codeWord"-> "PP1952")}
Classifiers
Classifiers ensure that the dependency being loaded is compatible with the platform for which the project is written. For example, to fetch the dependency relevant to JDK 1.5, use the following:
libraryDependencies += "org.testng" % "testng" % "5.7" classifier "jdk15"
We could also have multiple classifiers, as follows:
libraryDependencies += "org.lwjgl.lwjgl" % "lwjgl-platform" % lwjglVersion classifier "natives-windows" classifier "natives-linux" classifier "natives-osx"
Transitivity
In logic and mathematics, a relationship between three elements is said to be transitive. If the relationship holds between the first and second elements and between the second and third elements, it implies that it also holds a relationship between the first and third elements.
Relating this to the dependencies of a project, imagine that you have a project that depends on the project Foo for some of its functionality. Now, Foo depends on another project, Bar, for some of its functionality. If a change in the project Bar affects your project’s functionality, then this implies that your project indirectly depends on project Bar. This means that your project has a transitive dependency on the project Bar.
But if in this case a change in the project Bar does not affect your project’s functionality, then your project does not depend on the project Bar. This means that your project does not have a dependency on the project Bar.
SBT cannot know whether your project has a transitive dependency or not, so to avoid dependency issues, it loads the library dependencies transitively by default. In situations where this is not required for your project, you can disable it using intransitive() or notTransitive(). A common case where artifact dependencies are not required is in projects using the Felix OSGI framework (only its main JAR is required). The dependency can be declared as follows:
libraryDependencies += "org.apache.felix" % "org.apache.felix.framework" % "1.8.0" intransitive()
Or, it can be declared as follows:
libraryDependencies += "org.apache.felix" % "org.apache.felix.framework" % "1.8.0" notTransitive()
If we need to exclude certain transitive dependencies of a dependency, we could use the excludeAll or exclude method.
libraryDependencies += "log4j" % "log4j" % "1.2.15" exclude("javax.jms", "jms")libraryDependencies += "log4j" % "log4j" % "1.2.15" excludeAll( ExclusionRule(organization = "com.sun.jdmk"), ExclusionRule(organization = "com.sun.jmx"), ExclusionRule(organization = "javax.jms") )
Although excludeAll provides more flexibility, it should not be used in projects that will be published in the Maven style as it cannot be represented in a pom.xml file. The exclude method is more useful in projects that require pom.xml when being published, since it requires both organizationID and name to exclude a module.
Download documentation
Generally, an IDE plugin is used to download the source and API documentation JAR files. However, one can configure SBT to download the documentation without using an IDE plugin.
To download the dependency’s sources, add withSources() to the dependency definition. For example:
libraryDependencies += "org.apache.felix" % "org.apache.felix.framework" % "1.8.0" withSources()
To download API JAR files, add withJavaDoc() to the dependency definition. For example:
libraryDependencies += "org.apache.felix" % "org.apache.felix.framework" % "1.8.0" withSources() withJavadoc()
The documentation downloaded like this is not transitive. You must use the update-classifiers task to do so.
Dependencies using Maven files
SBT can be configured to use a Maven POM file to handle the project dependencies by using the externalPom method. The following statements can be used in the build definition:
- externalPom(): This will set pom.xml in the project’s base directory as the source for project dependencies
- externalPom(baseDirectory{base=>base/”myProjectPom”}): This will set the custom-named POM file myProjectPom.xml in the project’s base directory as the source for project dependencies
There are a few restrictions with using a POM file, as follows:
- It can be used only for configuring dependencies.
- The repositories mentioned in POM will not be considered. They need to be specified explicitly in the build definition or in an Ivy settings file.
- There is no support for relativePath in the parent element of POM and its existence will result in an error.
Dependencies using Ivy files or Ivy XML
Both Ivy settings and dependencies can be used to configure project dependencies in SBT through the build definition. They can either be loaded from a file or can be given inline in the build definition.
The Ivy XML can be declared as follows:
ivyXML := <dependencies> <dependency org="org.specs2" name="specs2" rev="1.12.3"></dependency></ dependencies>
The commands to load from a file are as follows:
- externalIvySettings(): This will set ivysettings.xml in the project’s base directory as the source for dependency settings.
- externalIvySettings(baseDirectory{base=>base/”myIvySettings”}): This will set the custom-named settings file myIvySettings.xml in the project’s base directory as the source for dependency settings.
- externalIvySettingsURL(url(“settingsURL”)): This will set the settings file at settingsURL as the source for dependency settings.
- externalIvyFile(): This will set ivy.xml in the project’s base directory as the source for dependency.
- externalIvyFile(baseDirectory(_/”myIvy”): This will set the custom-named settings file myIvy.xml in the project’s base directory as the source for project dependencies.
When using Ivy settings and configuration files, the configurations need to be mapped, because Ivy files specify their own configurations. So, classpathConfiguration must be set for the three main configurations. For example:
- classpathConfiguration in Compile := Compile
- classpathConfiguration in Test := Test
- classpathConfiguration in Runtime := Runtime
Adding JAR files manually
To handle dependencies manually in SBT, you need to create a lib folder in the project and add the JAR files to it. That is the default location where SBT looks for unmanaged dependencies. If you have the JAR files located in some other folder, you could specify that in the build definition. The key used to specify the source for manually added JAR files is unmanagedBase. For example, if the JAR files your project depends on are in project/extras/dependencies instead of project/lib, modify the value of unmanagedBase as follows:
unmanagedBase <<= baseDirectory {base => base/"extras/dependencies"}
Here, baseDirectory is the project’s root directory.
unmanagedJars is a task which lists the JAR files from the unmanagedBase directory. To see the list of JAR files in the interactive shell type, type the following:
> show unmanaged-jars [info] ArrayBuffer()
Or in the project folder, type:
$ sbt show unmanaged-jars
If you add a Spring JAR (org.springframework.aop-3.0.1.jar) to the dependencies folder, then the result of the previous command would be:
> show unmanaged-jars [info] ArrayBuffer(Attributed(/home/introduction/extras/dependencies/ org.springframework.aop-3.0.1.jar))
It is also possible to specify the path(s) of JAR files for different configurations using unmanagedJars. In the build definition, the unmanagedJars task may need to be replaced when the jars are in multiple directories and other complex cases.
unmanagedJars in Compile += file("/home/downloads/ org.springframework.aop-3.0.1.jar")
Resolvers
Resolvers are alternate resources provided for the projects on which there is a dependency. If the specified project’s JAR is not found in the default repository, these are tried. The default repository used by SBT is Maven2 and the local Ivy repository.
The simplest ways of adding a repository are as follows:
- resolvers += name at location.
For example:
resolvers += "releases" at "http://oss.sonatype.org/content/ repositories/releases"
- resolvers ++= Seq (name1 at location1, name2 at location2).
For example:
resolvers ++= Seq("snapshots" at "http://oss.sonatype.org/ content/repositories/snapshots", "releases" at "http://oss.sonatype.org/content/repositories/releases")
- resolvers := Seq (name1 at location1, name2 at location2).
For example:
resolvers := Seq("sgodbillon" at "https://bitbucket.org/ sgodbillon/repository/raw/master/snapshots/", "Typesafe backup repo" at " http://repo.typesafe.com/typesafe/repo/", "Maven repo1" at "http://repo1.maven.org/") )
You can also add their own local Maven repository as a resource using the following syntax:
resolvers += "Local Maven Repository" at "file://"+Path.userHome.absolutePath+"/.m2/repository"
An Ivy repository of the file types URL, SSH, or SFTP can also be added as resources using sbt.Resolver. Note that sbt.Resolver is a class with factories for interfaces to Ivy repositories that require a hostname, port, and patterns. Let’s see how to use the Resolver class.
For filesystem repositories, the following line defines an atomic filesystem repository in the test directory of the current working directory:
resolvers += Resolver.file ("my-test-repo", file("test")) transactional()
For URL repositories, the following line defines a URL repository at http://example.org/repo-releases/:
resolvers += Resolver.url(" my-test-repo", url("http://example.org/repo-releases/"))
The following line defines an Ivy repository at http://joscha.github.com/play-easymail/repo/releases/:
resolvers += Resolver.url("my-test-repo", url("http://joscha.github.com/play-easymail/repo/releases/")) (Resolver.ivyStylePatterns)
For SFTP repositories, the following line defines a repository that is served by SFTP from the host example.org:
resolvers += Resolver.sftp(" my-sftp-repo", "example.org")
The following line defines a repository that is served by SFTP from the host example.org at port 22:
resolvers += Resolver.sftp("my-sftp-repo", "example.org", 22)
The following line defines a repository that is served by SFTP from the host example.org with maven2/repo-releases/ as the base path:
resolvers += Resolver.sftp("my-sftp-repo", "example.org", "maven2/repo-releases/")
For SSH repositories, the following line defines an SSH repository with user-password authentication:
resolvers += Resolver.ssh("my-ssh-repo", "example.org") as("user", "password")
The following line defines an SSH repository with an access request for the given user. The user will be prompted to enter the password to complete the download.
resolvers += Resolver.ssh("my-ssh-repo", "example.org") as("user")
The following line defines an SSH repository using key authentication:
resolvers += { val keyFile: File = ... Resolver.ssh("my-ssh-repo", "example.org") as("user", keyFile, "keyFilePassword") }
The next line defines an SSH repository using key authentication where no keyFile password is required to be prompted for before download:
resolvers += Resolver.ssh("my-ssh-repo", "example.org") as("user", keyFile)
The following line defines an SSH repository with the permissions. It is a mode specification such as chmod:
resolvers += Resolver.ssh("my-ssh-repo", "example.org") withPermissions("0644")
SFTP authentication can be handled in the same way as shown for SSH in the previous examples.
Ivy patterns can also be given to the factory methods. Each factory method uses a Patterns instance which defines the patterns to be used. The default pattern passed to the factory methods gives the Maven-style layout. To use a different layout, provide a Patterns object describing it. The following are some examples that specify custom repository layouts using patterns:
resolvers += Resolver.url("my-test-repo", url)( Patterns("[organisation]/[module]/ [revision]/[artifact].[ext]") )
You can specify multiple patterns or patterns for the metadata and artifacts separately.
For filesystem and URL repositories, you can specify absolute patterns by omitting the base URL, passing an empty patterns instance, and using Ivy instances and artifacts.
resolvers += Resolver.url("my-test-repo") artifacts "http://example.org/[organisation]/[module]/ [revision]/[artifact].[ext]"
When you do not need the default repositories, you must override externalResolvers. It is the combination of resolvers and default repositories. To use the local Ivy repository without the Maven repository, define externalResolvers as follows:
externalResolvers <<= resolvers map { rs => Resolver.withDefaultResolvers(rs, mavenCentral = false) }
Summary
In this article, we have seen how dependency management tools such as Maven and Ivy work and how SBT handles project dependencies. This article also talked about the different options that SBT provides to handle your project dependencies and configuring resolvers for the module on which your project has a dependency.
Resources for Article :
Further resources on this subject:
- So, what is Play? [Article]
- Play! Framework 2 – Dealing with Content [Article]
- Integrating Scala, Groovy, and Flex Development with Apache Maven [Article]
