Home Tutorials Exploring the Nmap Scripting Engine API and Libraries

Exploring the Nmap Scripting Engine API and Libraries

February 13, 2015 - 12:00 am

4481

23 min read

This article written by Paulino Calderón Pale, the author of Mastering the Nmap Scripting Engine, teaches us about the usage of the most important NSE libraries.

This article explores the Nmap Scripting Engine API.

(For more resources related to this topic, see here.)

The NSE API and libraries allow developers to obtain host and port information, including versions of services, and perform a wide range of tasks when scanning networks with Nmap. As in any other programming language or framework, NSE libraries separate and refactor code that will likely be helpful for other NSE scripts. Tasks such as creating a network socket connection, storing valid credentials, or reading script arguments from the command line are commonly handled by these libraries. Nmap currently distributes 107 NSE libraries officially to communicate with the most popular protocols, perform common string handling operations, and even provide implementation classes such as the brute library, which provides a Driver class to quickly write your own password-auditing scripts.

This article covers the following topics:

Understanding the structure of an NSE script
Exploring the Nmap API and libraries
Sharing information between scripts with the NSE registry
Writing your own NSE libraries
Expanding the functionality of NSE libraries

After finishing this article, you will understand what information can be accessed through the Nmap API and how to update this information to reflect script results. My goal is to get you familiar with some of the most popular NSE libraries and teach you how to expand their functionality if needed.

Understanding the structure of an NSE script

An NSE script requires at least the following fields:

Description: This description is read by the –script-help Nmap option and is used in the documentation.
Categories: This field defines the script category used when selecting scripts. For a list of available categories, see Appendix C, Script Categories.
Action: This is the main function of the NSE script that gets executed on selection.
Execution rule: This defines when the script is going to run.

Other NSE script fields

Other available fields describe topics such as licensing, dependencies, and categories. These fields are optional, but I highly encourage you to add them to improve the quality of your script’s documentation.

Author

This field gives credits to the authors of the scripts who share their work with the community. It is acceptable to include e-mail addresses.

License

Developers are free to use whatever license they prefer but, if they would like to share their scripts and include them with official releases, they must use either Nmap’s licenses or licenses of the Berkeley Software Distribution (BSD) style.

The documentation describing Nmap’s license can be found at http://nmap.org/book/man-legal.html#nmap-copyright.

Dependencies

This field describes the possible dependencies between NSE scripts. This is useful when scripts require to be run in a specific order so that they can use the output of a previous script in another script. The scripts listed in the dependencies field will not run automatically, and they still require to be selected to run.

A sample NSE script

A simple NSE script looks like the following:

description = [[
Detailed description goes here
]]
---
-- @output
-- Some sample output
 
author = "Paulino Calderon <[email protected]>"
license = "Same as Nmap--See http://nmap.org/book/man-legal.html"
categories = {"discovery", "safe"}
 
-- Script is executed for any TCP port.
portrule = function( host, port )
  return port.protocol == "tcp"
end
 
--- main function
action = function( host, port )
  …
end

Exploring environment variables

There are a few environment variables that you need to consider when writing scripts because they will be helpful:

SCRIPT_PATH: This returns the absolute path of the running script
SCRIPT_NAME: This returns the running script name
SCRIPT_TYPE: This returns “prerule”, “hostrule”, “portrule”, or “postrule”

Use the SCRIPT_NAME environment variable instead of hardcoding the name of your script. This way, you won’t need to update the script if you end up changing its name. For example, you could use it to read script arguments as follows:

local arg1 = stdnse.get_script_args(SCRIPT_NAME..”.arg1″)

The stdnse library will be explored later in this article. This library contains the get_script_args() function that can be used to read script arguments.

Accessing the Nmap API

This is the core API that allows scripts to obtain host and port information such as name resolution, state, version detection results, Mac address, and more (if available). It also provides the interface to Nsock, Nmap’s socket library.

NSE arguments

The arguments passed to the main action function consist of two Lua tables corresponding to host and port information. The amount of information available depends on the options used during the scans. For example, the host.os table will show nothing if the OS detection mode (-O) was not set.

Host table

The host table is a regular Lua table with the following fields:

host.os: This is the table containing OS matches (only available with OS detection)
host.ip: This is the IP address of the target
host.name: This is the reverse DNS name of the target (if available)
host.targetname: This is the hostname specified in the command line
host.directly_connected: This is a Boolean that indicates whether the target is on the same network segment
host.mac_addr: This is the Mac address of the target
host.mac_addr_next_hop: This is the Mac address of the first hop to the target
host.mac_addr_src: This is the Mac address of our client
host.interface_mtu: This is the MTU value of your network interface
host.bin_ip: This is the target IP address as a 4-byte and 16-byte string for IPv4 and Ipv6, respectively
host.bin_ip_src: This is our client’s IP address as a 4-byte and 16-byte string for IPv4 and Ipv6, respectively
host.times: This is the timing data of the target
host.traceroute: This is only available with –traceroute

Port table

The port table is stored as a Lua table and it may contain the following fields:

port.number: This is the number of the target port.
port.protocol: This is the protocol of the target port. It could be tcp or udp.
port.service: This is the service name detected via port matching or with service detection (-sV).
port.version: This is the table containing the version information discovered by the service detection scan. The table contains fields such as name, name_confidence, product, version, extrainfo, hostname, ostype, devicetype, service_tunnel, service_ftp, and cpe code.
port.state: This returns information about the state of the port.

Exception handling in NSE scripts

The exception handling mechanism in NSE was designed to help with networking I/O tasks. It works in a pretty straightforward manner. Developers must wrap the code they want to monitor for exceptions inside an nmap.new_try() call. The first value returned by the function indicates the completion status. If it returns false or nil, the second returned value must be an error string. The rest of the return values in a successful execution can be set and used in any way.

The catch function defined by nmap.new_try() will execute when an exception is raised. Let’s look at the mysql-vuln-cve2012-2122.nse script (http://nmap.org/nsedoc/scripts/mysql-vuln-cve2012-2122.html). In this script, a catch function performs some simple garbage collection if a socket is left opened:

local catch = function() socket:close() end
local try = nmap.new_try(catch)
…
try( socket:connect(host, port) )
response = try( mysql.receiveGreeting(socket) )

The official documentation can be found at http://nmap.org/nsedoc/lib/nmap.html.

The NSE registry

The NSE registry is a Lua table designed to store variables shared between all scripts during a scan. The registry is stored at the nmap.registry variable. For example, some of the brute-force scripts will store valid credentials so that other scripts can use them to perform authenticated actions. We insert values as in any other regular Lua table:

table.insert( nmap.registry.credentials.http, { username =
username, password = password } )

Remember to select unique registry names to avoid overriding values used by other scripts.

Writing NSE libraries

When writing your own NSE scripts, you will sometimes want to refactor the code and make it available for others. The process of creating NSE libraries is pretty simple, and there are only a few things to keep in mind. NSE libraries are mostly in Lua, but other programming languages such as C and C++ can also be used.

Let’s create a simple Lua library to illustrate how easy it is. First, remember that NSE libraries are stored in the /nselib/ directory in your Nmap data directory by default. Start by creating a file named myfirstlib.lua inside it. Inside our newly written file, place the following content:

local stdnse = require "stdnse"
function hello(msg, name)
return stdnse.format("Hello '%s',n%s", msg, name)
end

The first line declares the dependency with the stdnse NSE library, which stores useful functions related to input handling:

local stdnse = require "stdnse"
The rest is a function declaration that takes two arguments and passes them through the stdnse library's format function:
function hello(msg, name)
  return stdnse.format("Hello '%s',n%s", msg, name)
end

Now we can call our new library from any script in the following way:

local myfirstlib = require "myfirstlib"
…
myfirstlib.hello("foo", "game over!")
…

Remember that global name collision might occur if you do not choose meaningful names for your global variables.

The official online documentation of the stdnse NSE library can be found at http://nmap.org/nsedoc/lib/stdnse.html

Extending the functionality of an NSE library

The available NSE libraries are powerful and comprehensive but, sometimes, we will find ourselves needing to modify them to achieve special tasks. For me, it was the need to simplify the password-auditing process that performs word list mangling with other tools, and then running the scripts in the brute category. To simplify this, let’s expand the functionality of one of the available NSE libraries and a personal favorite: the brute NSE library. In this implementation, we will add a new execution mode called pass-mangling, which will perform common password permutations on-the-fly, saving us the trouble of running third-party tools.

Let’s start to write our new iterator function. This will be used in our new execution mode. In our new iterator, we define the following mangling rules:

digits: Appends common digits found in passwords such as single- and double-digit numbers and common password combinations such as 123
strings: Performs common string operations such as reverse, repetition, capitalization, camelization, leetify, and so on
special: Appends common special characters such as !, $, #, and so on
all: This rule executes all the rules described before

For example, the word secret will yield the following login attempts when running our new brute mode pass-mangling:

secret2014
secret2015
secret2013
secret2012
secret2011
secret2010
secret2009
secret0
secret1
secret2
...
secret9
secret00
secret01
...
secret99
secret123
secret1234
secret12345
s3cr3t
SECRET
S3CR3T
secret
terces
Secret
S3cr3t
secretsecret
secretsecretsecret
secret$
secret#
secret!
secret@

Our new iterator function, pw_mangling_iterator, will take care of generating the permutations corresponding to each rule. This is a basic set of rules that only takes care of common password permutations. You can work on more advanced password-mangling rules after reading this:

pw_mangling_iterator = function( users, passwords, rule)
  local function next_credential ()
    for user, pass in Iterators.account_iterator(users, passwords, 
 "pass") do
      if rule == 'digits' or rule == 'all' then
        -- Current year, next year, 5 years back...
        local year = tonumber(os.date("%Y"))
        coroutine.yield( user, pass..year )
        coroutine.yield( user, pass..year+1 )
        for i = year, year-5, -1 do
          coroutine.yield( user, pass..i )
        end
 
        -- Digits from 0 to 9
        for i = 0, 9 do
          coroutine.yield( user, pass..i )
        end
        -- Digits from 00 to 99
        for i = 0, 9 do
          for x = 0, 9 do
            coroutine.yield( user, pass..i..x )
          end
        end
 
        -- Common digit combos
        coroutine.yield( user, pass.."123" )
        coroutine.yield( user, pass.."1234" )
        coroutine.yield( user, pass.."12345" )
      end
      if rule == 'strings' or rule == 'all' then
        -- Basic string stuff like uppercase,
        -- reverse, camelization and repetition
        local leetify = {["a"] = '4',
                         ["e"] = '3',
                         ["i"] = '1',
                         ["o"] = '0'}
        local leetified_pass = pass:gsub("%a", leetify)
        coroutine.yield( user, leetified_pass )
        coroutine.yield( user, pass:upper() )
        coroutine.yield( user, leetified_pass:upper() )
        coroutine.yield( user, pass:lower() )
        coroutine.yield( user, pass:reverse() )
        coroutine.yield( user, pass:sub(1,1):upper()..pass:sub(2) 
 )
        coroutine.yield( user,    
 leetified_pass:sub(1,1):upper()..leetified_pass:sub(2) )
        coroutine.yield( user, pass:rep(2) )
        coroutine.yield( user, pass:rep(3) )
      end
      if rule == 'special' or rule == 'all' then
        -- Common special characters like $,#,!
        coroutine.yield( user, pass..'$' )
        coroutine.yield( user, pass..'#' )
        coroutine.yield( user, pass..'!' )
        coroutine.yield( user, pass..'.' )
        coroutine.yield( user, pass..'@' )
      end
    end
    while true do coroutine.yield(nil, nil) end
  end
  return coroutine.wrap( next_credential )
end

We will add a new script argument to define the brute rule inside the start function of the brute engine:

local mangling_rules = stdnse.get_script_args("brute.mangling-
 rule") or "all"

In this case, we also need to add an elseif clause to execute our mode when the pass-mangling string is passed as the argument. The new code block looks like this:

…

    elseif( mode and mode == 'pass' ) then
      self.iterator = self.iterator or Iterators.pw_user_iterator( 
 usernames, passwords )
    elseif( mode and mode == 'pass-mangling' ) then
      self.iterator = self.iterator or 
 Iterators.pw_mangling_iterator( usernames, passwords, 
 mangling_rules )
    elseif ( mode ) then
      return false, ("Unsupported mode: %s"):format(mode)

…

With this simple addition of a new iterator function, we have inevitably improved over 50 scripts that use this NSE library. Now you can perform password mangling on-the-fly for all protocols and applications. At this point, it is very clear why code refactoring in NSE is a major advantage and why you should try to stick to the available implementations such as the Driver brute engine.

NSE modules in C/C++

Some modules included with NSE are written in C++ or C. These languages provide enhanced performance but are only recommended when speed is critical or the C or C++ implementation of a library is required.

Let’s build an example of a simple NSE library in C to get you familiar with this process. In this case, our C module will contain a method that simply prints a message on the screen. Overall, the steps to get a C library to communicate to NSE are as follows:

Place your source and header files for the library inside Nmap’s root directory
Add entries to the source, header, and object file for the new library in the Makefile.in file
Link the new library from the nse_main.cc file

First, we will create our library source and header files. The naming convention for C libraries is the library name appended to the nse_ string. For example, for our library test, we will name our files nse_test.cc and nse_test.h. Place the following content in a file named nse_test.cc:

extern "C" {
  #include "lauxlib.h"
  #include "lua.h"
}
 
#include "nse_test.h"
 
static int hello_world(lua_State *L) {
  printf("Hello World From a C libraryn");
  return 1;
}
 
static const struct luaL_Reg testlib[] = {
  {"hello",    hello_world},
  {NULL, NULL}
};
 
LUALIB_API int luaopen_test(lua_State *L) {
  luaL_newlib(L, testlib);
  return 1;
}

Then place this content in the nse_test.h library header file:

#ifndef TESTLIB
#define TESTLIB
 
#define TESTLIBNAME "test"
 
LUALIB_API int luaopen_test(lua_State *L);
 
#endif

Make the following modifications to the nse_main.cc file:

Include the library header at the beginning of the file:
```
#include <nse_test.h>
```

Look for the set_nmap_libraries(lua_State *L) function and update the libs variable to include the new library:

static const luaL_Reg libs[] = {
    {NSE_PCRELIBNAME, luaopen_pcrelib},
    {NSE_NMAPLIBNAME, luaopen_nmap},
    {NSE_BINLIBNAME, luaopen_binlib},
    {BITLIBNAME, luaopen_bit},
    {TESTLIBNAME, luaopen_test},
    {LFSLIBNAME, luaopen_lfs},
    {LPEGLIBNAME, luaopen_lpeg},
#ifdef HAVE_OPENSSL
    {OPENSSLLIBNAME, luaopen_openssl},
#endif
    {NULL, NULL}
  };

Add the NSE_SRC, NSE_HDRS, and NSE_OBJS variables to Makefile.in:

NSE_SRC=nse_main.cc nse_utility.cc nse_nsock.cc nse_dnet.cc 
 nse_fs.cc nse_nmaplib.cc nse_debug.cc nse_pcrelib.cc 
 nse_binlib.cc nse_bit.cc nse_test.cc nse_lpeg.cc
NSE_HDRS=nse_main.h nse_utility.h nse_nsock.h nse_dnet.h 
 nse_fs.h nse_nmaplib.h nse_debug.h nse_pcrelib.h 
 nse_binlib.h nse_bit.h nse_test.h nse_lpeg.h
NSE_OBJS=nse_main.o nse_utility.o nse_nsock.o nse_dnet.o 
 nse_fs.o nse_nmaplib.o nse_debug.o nse_pcrelib.o 
 nse_binlib.o nse_bit.o nse_test.o nse_lpeg.o

Now we just need to recompile and create a sample NSE script to test our new library.

Create a file named nse-test.nse inside your scripts folder with the following content:

local test = require "test"
 
description = [[
Test script that calls a method from a C library
]]
 
author = "Paulino Calderon <calderon()websec.mx>"
license = "Same as Nmap--See http://nmap.org/book/man-legal.html"
categories = {"safe"}
 
 
portrule = function() return true end
 
action = function(host, port)
        local c = test.hello()
end

Finally, we execute our script. In this case, we will see the Hello World From a C library message when the script is executed:

$nmap -p80 --script nse-test scanme.nmap.org
Starting Nmap 6.47SVN ( http://nmap.org ) at 2015-01-13 23:41 
 CST
Hello World From a C library
Nmap scan report for scanme.nmap.org (74.207.244.221)
Host is up (0.12s latency).
PORT   STATE SERVICE
80/tcp open  http
 Nmap done: 1 IP address (1 host up) scanned in 0.79 seconds

To learn more about Lua’s C API and how to run compiled C modules, check out the official documentation at http://www.lua.org/manual/5.2/manual.html#4 and http://nmap.org/book/nse-library.html

Exploring other popular NSE libraries

Let’s briefly review some of the most common libraries that you will likely need during the development of your own scripts. There are 107 available libraries at the moment, but the following libraries must be remembered at all times when developing your own scripts in order to improve their quality.

stdnse

This library contains miscellaneous functions useful for NSE development. It has functions related to timing, parallelism, output formatting, and string handling.

The functions that you will most likely need in a script are as follows:

stdnse.get_script_args: This gets script arguments passed via the –script-args option:
```
local threads = 
 stdnse.get_script_args(SCRIPT_NAME..".threads") or 3
```

stdnse.debug: This prints a debug message:

stdnse.debug2("This is a debug message shown for debugging 
 level 2 or higher")

stdnse.verbose: This prints a formatted verbosity message:
```
stdnse.verbose1("not running for lack of privileges.")
```
stdnse.strjoin: This joins a string with a separator string:
```
local output = stdnse.strjoin("n", output_lines)
```
stdnse.strsplit: This splits a string by a delimiter:
```
local headers = stdnse.strsplit("rn", headers)
```

The official online documentation of the stdnse NSE library can be found at http://nmap.org/nsedoc/lib/stdnse.html

openssl

This is the interface to the OpenSSL bindings used commonly in encryption, hashing, and multiprecision integers. Its availability depends on how Nmap was built, but we can always check whether it’s available with some help of a pcall() protected call:

if not pcall(require, "openssl") then
  action = function(host, port)
    stdnse.print_debug(2, "Skipping "%s" because OpenSSL is 
 missing.", id)
  end
end
action = action or function(host, port)
  ...
end

The official online documentation of the openssl NSE library can be found at http://nmap.org/nsedoc/lib/openssl.html

target

This is a utility library designed to manage a scan queue of newly discovered targets. It enables NSE scripts running with prerule, hostrule, or portrule execution rules to add new targets to the current scan queue of Nmap on-the-fly. If you are writing an NSE script belonging to the discovery category, I encourage you to use this library in the script.

To add targets, simply call the target.add function:

local status, err = target.add("192.168.1.1","192.168.1.2",...)

The official online documentation of the target NSE library can be found at http://nmap.org/nsedoc/lib/target.html

shortport

This library is designed to help build port rules. It attempts to collect in one place the most common port rules used by script developers. To use it, we simply load the library and assign the corresponding port rule:

local shortport = require "shortport"
…
portrule = shortport.http

The most common functions that you are likely to need are as follows:

http: This is the port rule to match HTTP services:
```
portrule = shortport.http
```

port_or_service: This is the port rule to match a port number or service name:

portrule = shortport.port_or_service(177, "xdmcp", "udp")

portnumber: This is the port rule to match a port or a list of ports:

portrule = shortport.portnumber(69, "udp")

The official online documentation of the shortport NSE library can be found at http://nmap.org/nsedoc/lib/shortport.html

creds

This library manages credentials found by the scripts. It simply stores the credentials in the registry, but it provides a clean interface to work with the database.

To add credentials to the database, you simply need to create a creds object and call the add function:

local c = creds.Credentials:new( SCRIPT_NAME, host, port )
  c:add("packtpub", "secret", creds.State.VALID )

The official online documentation of the creds NSE library can be found at http://nmap.org/nsedoc/lib/creds.html.

vulns

This library is designed to help developers present the state of a host with regard to security vulnerabilities. It manages and presents consistent and human-readable reports for every vulnerability found in the system by NSE. A report produced by this library looks like the following:

PORT   STATE SERVICE REASON
80/tcp open  http    syn-ack
http-phpself-xss:
   VULNERABLE:
   Unsafe use of $_SERVER["PHP_SELF"] in PHP files
     State: VULNERABLE (Exploitable)
     Description:
       PHP files are not handling safely the variable 
 $_SERVER["PHP_SELF"] causing Reflected Cross Site Scripting 
 vulnerabilities.
             
     Extra information:
      
   Vulnerable files with proof of concept:
     http://calder0n.com/sillyapp/three.php/%27%22/%3E%3Cscript%3Ealert
 (1)%3C/script%3E
     http://calder0n.com/sillyapp/secret/2.php/%27%22/%3E%3Cscript%3Eal
 ert(1)%3C/script%3E
     http://calder0n.com/sillyapp/1.php/%27%22/%3E%3Cscript%3Ealert(1)%
 3C/script%3E
     http://calder0n.com/sillyapp/secret/1.php/%27%22/%3E%3Cscript%3Eal
 ert(1)%3C/script%3E
   Spidering limited to: maxdepth=3; maxpagecount=20; 
 withinhost=calder0n.com
     References:
       https://www.owasp.org/index.php/Cross-site_Scripting_(XSS)
      http://php.net/manual/en/reserved.variables.server.php

The official online documentation of the vulns NSE library can be found at http://nmap.org/nsedoc/lib/vulns.html.

http

Nmap has become a powerful Web vulnerability scanner, and most of the tasks related to HTTP can be done with this library. The library is simple to use, allows raw header handling, and even has support to HTTP pipelining.

It has methods such as http.head(), http.get(), and http.post(), corresponding to the common HTTP methods HEAD, GET, and POST, respectively, but it also has a generic method named http.generic_request() to provide more flexibility for developers who may want to try more obscure HTTP verbs.

A simple HTTP GET call can be made with a single method call:

local respo = http.get(host, port, uri)

The official online documentation of the http NSE library can be found at http://nmap.org/nsedoc/lib/http.html.

Summary

In this article, you learned what information is available to NSE and how to work with this data to achieve different tasks with Nmap. You also learned how the main NSE API works and what the structures of scripts and libraries are like. We covered the process of developing new NSE libraries in C and Lua. Now you should have all of the knowledge in Lua and the inner workings of NSE required to start writing your own scripts and libraries.

Resources for Article:

Further resources on this subject: