Iterating over a CSV file in JMeter

A common problem in JMeter is that you want a CSV file as an input and want to execute each line, then continue. So from a script perspective that would look something like this:

Thread Group
 |-For Each CSV Line
 |  |-HTTP Request (CSV input)
 |-Do something else

The issue though is there is no “For Each” concept in JMeter. I know that it is called a ForEach Loop in JMeter but it isn’t from a developer view. You need to know the length of your CSV file. You could just tell the For loop how many lines there are but that would make it very unflexible. You’d need to adjust the script every time the CSV changes.

So here is how I solved it. Not elegant but it actually works quite well.

I have a CSV file that looks like this (the content is kept simple on purpose! 😉 ):

 1,/
 2,/?page=1
 3,/?page=2

My script looks like this:

The Site we’re testing is http://slashdot.org, which is postfixed by the URL in the CSV file. So we should have the ForEach loop iterate three times. Here is the explanation on how it works.

User Parameters 1 contains GETfile=filename.csv

This gets posted to the GetLines JSR223 controller running a Groovy script.

String Filename = vars.get("GETfile")
String fileContents = new File(Filename).getText('UTF-8')
def lines = 0  
    fileContents.eachLine { line ->
        lines++
        vars.put("CSVLine_" + lines, line)
    }
vars.put("GETfileLength",lines.toString())   
log.debug(Filename + " has " + lines + " lines.")

The Groovy scripts outputs two variables:

  1. GETfileLenght, which contains the number of lines of the CSV file
  2. CSVLine_XXX, which are enumerated variables (XXX) that contain the contents of each line.

The ForEach Loop now looks like this:

jmeterhtw_ForEach

Note that the start of the index is zero although values start at 1! So the loop will iterate over CSVLine_1, CSVLine_2 and CSVLine_3. Each time the line content is placed into LineContent. Butthe data is still just a CSV line, so we must split it to get to the separate items in the line. This is, what User Parameters 2 does.

User Parameters 2 contains CSVLineSplit=${__split(${LineContent},CSVLineSplit)}

This splits the line into two variables. For the 1st line that would be:

CSVLineSplit_1=1
CSVLineSplit_2=/

Now we can put this into the HTTP Request.

 jmeterhtw_HTTPRequest

DONE!

Not the nicest of solutions but all others I tried were uglier and tinkering with <EOF> and While Loops doesn’t really work if you want more than one execution in a thread (most solutions I googled were of that type).

This scipt can also be extended to read the assertions from the CSV too. You could now feed a web server log into JMeter to execute or something like that.

 

by Oliver Erlewein

11 thoughts on “Iterating over a CSV file in JMeter

  1. Thanks for the tip, Oliver, but actually this scenario can be implemented much easier like:

    1. Set “Recycle on EOF” to “False” in CSV Data Set Config
    2. Add While Controller to your Test Plan
    3. Put samplers and CSV Data Set Config under the While Controller
    4. Use the following condition: ${__javaScript(“${foo}” != “”,)} (replace “foo” with the actual variable name)
    5. That’s it.

    See Using CSV DATA SET CONFIG guide for more detailed information on parametrising your JMeter test using external CSV data sources.

  2. Pingback: Testing Bits – 12/27/15 – 1/2/16 | Testing Curator Blog

  3. Thanks for the post Oliver. However I’m having a bit of a problem when I try and use this method on JMeter 3.0. when I look at my request results, ${LineSplit_1} is just giving me ‘${LineContent}’ rather than a value or the first column of the row. The only thing I can think of at the moment is that I used JSR223 sampler rather than a controller because I didn’t see that option anywhere. Any thoughts or ideas on this?

    • You most likely have a “spelling” mistake in there somewhere. I used the name CSVLineSplit. But I am actually thinking that ${LineContent} isn’t properly set. So either a spelling mistake or you should check your For Each controller and if it is as above.

      • Thanks for the reply but no alas I wish it were a spelling mistake that would have been easier to track. I tracked it down to the split function. for some reason in my debugger I saw that CSVLineSplit_1 was being set to ${lineContent} yet lineContent did have the row data from the CSV i.e a CSVLine_# variable from the script). So I knew that the for each controller was working. So to test this I disabled the ‘user parameters 2’ and added a script that hand rolls a split. Which isn’t pretty but that solved the issue. no idea why but for some reason it couldn’t resolve the variable reference from the for each controller (bug maybe? IDK). In any case here’s my ugo script in case anyone ever runs into this…

        String tempLine = vars.get(lineContent”) //<–LineContent in the example post
        log.debug("CSV tempLine: " + tempLine)
        String[] CSVLineSplit = tempLine.split(',')
        vars.put("CSVLineSplit_1", CSVLineSplit[0]) //repeat per CSV file column.
        vars.put("CSVLineSplit_2", CSVLineSplit[1])

        • That indeed is weird. Try raise it as a bug. I’m guessing you are running 3.1?
          Thanks for the fix suggestion. I must admit I haven’t run it on 3.x yet.

  4. Thanks for the solution Oliver. However i’m getting stuck due to an overlap of values. The scenario is to download different reports PDF of each ${Month} between the ${From} & ${To} variables. So the variables are getting captured as ${CSVLineSplit_1},${CSVLineSplit_2},${CSVLineSplit_3}

    But it seems like every time the loop is iterated the ${CSVLineSplit_1} value is not getting updated. The results in getting PDF report say of February month with From date as 1 May 2017.

    I’m thinking of inserting a JSR223 postprocessor in the PDF download HTTP request sampler to update the value of CSVLineSplit_1. Though not sure on this. Can you please advise.

  5. Hi,

    Thanks for the nice tutorial, looks very helpful. Anyhow i have another problem if applicable to be posted here. In the csv file i have couple of lines per (ex. 4) for each thread, and in the POST request i need to pass those couple of lines in an array. Is there solution for that? All of the answers that i found are just for one value from the file that is mapped in the request property.

    Example of the file:

    custId, number
    cvalue1, nvalue1
    cvalue1, nvalue2
    cvalue1, nvalue3
    cvalue1, nvalue4
    cvalue2, nvalue5
    cvalue2, nvalue6
    cvalue2, nvalue7
    cvalue2, nvalue8

    Example of the request:

    (first thread)
    POST: /cvalue1/
    {
    “numbers”: [
    {
    “number”: “${nvalue1}”
    },
    {
    “number”: “${nvalue2}”
    },
    {
    “number”: “${nvalue3}”
    },
    {
    “number”: “${nvalue4}”
    }
    ]
    }

    (second thread)
    POST: /cvalue2/
    {
    “numbers”: [
    {
    “number”: “${nvalue5}”
    },
    {
    “number”: “${nvalue6}”
    },
    {
    “number”: “${nvalue7}”
    },
    {
    “number”: “${nvalue8}”
    }
    ]
    }

    Thanks

    • Hi
      I’d probably try putting a loop around the CSV read step 1st. If that doesn’t work I’d end up coding it in Groovy relatively quickly. There you have more flexibility around reads. Might be worth looking at the JMeter Plugins too. Maybe you can find something pre-made there. If not might be a good suggestion for a plugin extension or a new plugin.
      Cheers Oliver

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s