Even more dependency files from your repos

The vast majority of Gemnasium users have the source code of their projects hosted on GitHub. Over the last months, many of them have asked for two things:

  • Many developers wanted Gemnasium to look for dependency files in subdirectory, outside of the root directory of the GitHub repo.
  • Python developers wanted the -r directive to be supported. It makes possible to include a Python dependency file from another one.

We are happy to announce that we now provide these two features!

Recursive dependency file search

Gemnasium can search for dependency files through the entire GitHub repository of your project. The recursive search is disabled by default; it can be enabled from the “Info” tab of the project settings page.

Why not enabling the recursive search by default? Well, it’s not so common to have dependency files in subdirectories. Besides, you probably have dependency files in vendored libs (especially javascript ones), you don’t want them to be parsed!

If your project is not hosted on GitHub (aka “offline” project), it was already possible to push any dependency file, no matter what directory it was in. As an example, let’s consider a Ruby On Rails project having its dependency files in a /gemfiles directory. Here is the command line to push these files to Gemnasium.com:

$ gemnasium df push --files=gemfiles/Gemfile,gemfiles/Gemfile.lock

By the way, don’t forget to push all your dependency files at a time!

If not familiar with the command line client, please have a look at the Gemnasium Toolbelt project.

Python -r support

Gemansium now detects the -r xxx pattern in Python dependency files, and fetches the file that is given as an argument. This was a feature awaited for very long time by the Python community.

How does this work? We’ve extended Gemnasium with pre-processors that analyse the content of each dependency file before it goes through the parsers (ie. before the dependencies are extracted). The Python pre-processor detects the -r option and adds the given path to the list of files to be fetched, processed and parsed. The included file goes through the pre-processor too, so it’s possible to include files recursively.

Now let’s consider you have a Python project with a file named requirement.txt in the root directory of the GitHub repo. Here is the content of the file:

pyramid >=1.0
SQLAlchemy >= 0.8.1
transaction == 1.4.1
pyramid_simpleform >=0.8
-r requirements/test.txt

Gemnasium will fetch and parse requirements/test.txt. Here is its content:

pyramid_debugtoolbar > 1.0.8

The included file can be in any subdirectory as long as the path matches requirements*.txt. So a path like test-requirements.txt would work fine too.

There’s no need to enable the “recursive dependency file search” if the files you include with -r are in subdirectories. Actually, it’s even better not to do so.

To know more about the -r option, have a look at the requirements file format.

GitHub Symlinks fix

Gemnasium now handles git symlinks properly when syncing with GitHub. This is another option if your dependency file are in a subdirectory and you want them to be parsed by Gemnasium.

For instance, let’s go back to the previous example: a Ruby On Rails project with two dependency files in a /gemfiles directories. Unless the recursive search is enabled, these files will be ignored. But we can create two symlinks to duplicate these files in the root directory:

# from the root directory of the repo:
$ ln -s gemfiles/Gemfile Gemfile
$ ln -s gemfiles/Gemfile.lock Gemfile.lock

To sum it up, you now have more options if your dependency files don’t match the usual names and directories. Your project still doesn’t fit? Please contact us!