NeuralNetTools 1.0.0 now on CRAN

After successfully navigating the perilous path of CRAN submission, I’m pleased to announce that NeuralNetTools is now available!  From the description file, the package provides visualization and analysis tools to aid in the interpretation of neural networks, including functions for plotting, variable importance, and sensitivity analyses. I’ve written at length about each of these functions (see here, here, and here), so I’ll only provide an overview in this post. Most of these functions have remained unchanged since I initially described them, with one important change for the Garson function. Rather than reporting variable importance as -1 to 1 for each variable, I’ve returned to the original method that reports importance as 0 to 1. I was getting inconsistent results after toying around with some additional examples and decided the original method was a safer approach for the package. The modified version can still be installed from my GitHub gist. The development version of the package is also available on GitHub. Please use the development page to report issues.

The package is fairly small but I think the functions that have been included can help immensely in evaluating neural network results. The main functions include:

  • plotnet: Plot a neural interpretation diagram for a neural network object, original blog post here
    # install, load package
    # create model
    AND <- c(rep(0, 7), 1)
    OR <- c(0, rep(1, 7))
    binary_data <- data.frame(expand.grid(c(0, 1), c(0, 1), c(0, 1)), AND, OR)
    mod <- neuralnet(AND + OR ~ Var1 + Var2 + Var3, binary_data,
                     hidden = c(6, 12, 8), rep = 10, err.fct = 'ce', linear.output = FALSE)
    # plotnet
    par(mar = numeric(4), family = 'serif')
    plotnet(mod, alpha = 0.6)
    Fig: Using the plotnet function.

  • garson: Relative importance of input variables in neural networks using Garson’s algorithm, original blog post here
    # create model
    x <- neuraldat[, c('X1', 'X2', 'X3')]
    y <- neuraldat[, 'Y1']
    mod <- mlp(x, y, size = 5)
    # garson
    garson(mod, 'Y1')
    Fig: Using the garson function.

  • lekprofile: Conduct a sensitivity analysis of model responses in a neural network to input variables using Lek’s profile method, original blog post here
    # create model
    mod <- nnet(Y1 ~ X1 + X2 + X3, data = neuraldat, size = 5)
    # lekprofile
    Fig: Using the lekprofile function.

A few other functions are available that are helpers to the main functions. See the documentation for a full list.

All the functions have S3 methods for most of the neural network classes available in R, making them quite flexible. This includes methods for nnet models from the nnet package, mlp models from the RSNNS package, nn models from the neuralnet package, and train models from the caret package. The functions also have methods for numeric vectors if the user prefers inputting raw weight vectors for each function, as for neural network models created outside of R.

Huge thanks to Hadley Wickham for his packages that have helped immensely with this process, namely devtools and roxygen2. I also relied extensively on his new web book for package development. Any feedback regarding NeuralNetTools or its further development is appreciated!




22 thoughts on “NeuralNetTools 1.0.0 now on CRAN

  1. I found 2 small mistakes at the beginning of the example code:
    install.packages(‘NeuralNetTools’) # quotes
    library(neuralnet) # load package to call function neuralnet()

    • Ah my mistake, these packages are installed with NeuralNetTools but not loaded. They’re not needed to use the package but of course you need them to create neural networks!

  2. Condensed News | Data Analytics & R

  3. Wonderful package, thanks!
    I think that it would be useful if the documentation said that the first value in each layer is the bias (I found this info somewhere in your blog).

    • Hi Kinetic, thanks and I’m glad you like the package. The documentation for the neuralweights function describes the order of the weights that are returned by the function. I’ve added some text to the help file to make this more clear. You can download the development version of this package to see the changes:


      The following was added to the neuralweights help file:

      For example, the elements in a list item for ‘hidden 1 1’ of a neural network with a 3 5 1 structure (3 inputs, 5 hidden nodes, 1 output) would have four values indicating the weights in sequence from the bias layer, first input layer, second input layer, and third input layer going to the first hidden node.

  4. I am writing a package that would allow me to fit a neural network on any acyclic graph structure. The underlying data structure is igraph. Could I get your package to work on an igraph or graphNEL data structure?

    • Hi Robert,

      I haven’t used igraph at all, but in hindsight, it looks like a more straightforward approach to plotting neural networks. The plotnet function in my package uses base graphics… kind of old school, though I certainly see the benefit of tapping into some of these newer packages like igraph. I suspect that the plotting functions in my package would have to be reworked quite a bit, but this would definitely be a neat addition. I’ll consider it for future updates, but it definitely will not work in the current version.


  5. Hi Marcus,

    This is very nice job!

    I am working with a package that will give me MLP Neural weights, but anything more than 1 Layer and I am having trouble passing the weights in the right structure for plotting and the other functions.

    In your example for using numeric input weights and structure, you only have 1 layer. Is it possible to explain how would one pass the wts_in for multiple layers?

    Thanks a lot, and congrats!

    • Hi Marcos, maybe this example helps? I created a neural network with multiple hidden layers, then extracted the weights to show how they would look for a numeric input. You will have to install the development version of the package for this to work. The names for the elements in the ‘wts’ object should provide some clues. You’ll also notice that the plot for the original model is identical (except for the labels) for the plot from the numeric inputs.

      ## devtools::install_github('fawda123/NeuralNetTools')
      ## create data for teh example
      AND <- c(rep(0, 7), 1)
      OR <- c(0, rep(1, 7))
      binary_data <- data.frame(expand.grid(c(0, 1), c(0, 1), c(0, 1)), AND, OR)
      # a neural network with two hidden layers, first has two nodes, second has three nodes
      mod <- neuralnet(AND + OR ~ Var1 + Var2 + Var3, binary_data, 
       hidden = c(2, 3), rep = 10, err.fct = 'ce', linear.output = FALSE)
      # get the weights and structure in the right format
      wts <- neuralweights(mod)
      struct <- wts$struct
      wts <- unlist(wts$wts)
      # plot
      plotnet(wts, struct = struct)
  6. Marcus,

    Thank you so much for this work. Last year I was happy to discover your plot.nnet code before teaching my students (at an R certificate program, University of Washington) neural nets. It was extremely useful.

    It’s even greater that you made it into a package now! Yes, from experience getting into CRAN *is* almost always worth the hassle.

    All the best, Assaf

    • Thanks Andreas, interesting paper. Am I correct in understanding that you looked only at the rank orders for each method? I think a descriptive advantage of Olden’s technique is the ability to describe both magnitude and sign of importance, whereas Garson’s only describes magnitude. It may also be interesting to add a random error component to each of your linear models. If I’m understanding correctly, it looks like you’ve made Y a linear combination of the X variables with no error component.

      • Hi Marcus,

        I looked at both rank orders (see Table 1) and mean-values (see Figure 1) and I examined both linear and nonlinear relations between Y and the X variables.

        I also tried adding error components as well as correlations between the X variables, but this did not change the results, so I decided to keep it simple: In case of non-monotonic effects (e.g., equations 2 and 3) magnitudes, signs and rank order of Olden’s coefficients were often wrong, whereas Garson’s coefficients adequately represented the relative importance of variables in each of the datasets simulated (i.e., even in case of non-monotonic relations).

  7. If I am trying to apply your olden-method to a nnet with only one hidden-unit, the following error occurs (whereas it works fine in case of more hidden-units):

    Error in rowSums(sum_in) :
    ‘x’ must be an array of at least two dimensions

  8. Hi Marcus,

    How do I use lekprofile method for the object that was created using h2o package?

    When I am trying to use lekprofile method as

    Error that I am receiving is:
    Error in is.finite(from) : default method not implemented for type ‘S4’

    I appreciate if you could please help me this regard.

    Many thanks

  9. Dear Sir,
    Your articles are very helpful for me. And, I have two output nodes, but garson only applies to neural networks with one output node. What should i do?

    • Hi, download the latest version of NeuralNetTools from CRAN and try using the ‘olden’ function. It’s similar to ‘garson’ but can evaluate more than one output node. You will have to change the ‘out_var’ argument which tells the function which output node to evaluate.


    • Hi Marcus,

      very nice package! I am installing it right now.

      Just a couple of question for you:
      1) as I understand it seems like that some functions are fine with multilayer networks but others (e.g. garson and lekprofile), are you planning to extend their functionalities in the next future? And,
      2) in regards to lekprofile, I think the function should allow multirow multi-plots depending on the number of explanatory variables considered. Or should give the possibility to the user to re-plot the output as he/she prefers. What do you think about this?

      Many thanks,


      • Hi Federico, glad you are finding the package useful. For your first question, the Garson method was developed to work with networks having only one layer. I don’t believe the original method was ever intended to be extended to multilayer networks. I remember exploring this a few years ago and not finding it possible. Personally, I think the olden method provides a more informative summary of variable importance and it can evaluate multilayer networks. The lekprofile should work fine with networks having more than one layer. For your second question, I’m not sure I follow exactly. If you are talking about giving the user flexibility in the layout of the plot, I’ve provided an argument in lekprofile that returns the raw values. These can be used to create your own plot, as an example:

        #define number of variables and observations
        #define correlation matrix for explanatory variables 
        #define actual parameter values
        y1<-rand.vars %*% matrix(parms1) + rnorm(num.obs,sd=20)
        y2<-rand.vars %*% matrix(parms2) + rnorm(num.obs,sd=20)
        #prep data and create neural network
        resp<-apply(cbind(y1,y2),2, function(y) (y-min(y))/(max(y)-min(y)))
        # get raw data from lekprofile
        rawdat <- lekprofile(mod1, val_out = T)[[1]]
        # make your own plot
        ggplot(rawdat, aes(x = Explanatory, y = Response, group = Groups, colour = Groups)) + 
          geom_line() + 
          facet_wrap(resp_name ~ exp_name)



      • Thanks Marcus,
        in fact I read the article by Garson and documentation of the package only after posting my comment. Sorry about that.
        Thanks a lot for your help and support with this!
        Good job!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s