Jkoder.com Tutorials, Tips and interview questions for Java, J2EE, Android, Spring, Hibernate, Javascript and other languages for software developers
This tutorial will demonstrate how to execute multiple curl requests in parallel, at the same time. We will use xargs command of linux or UNIX like operating system.
What does xargs command do in Linux?
xargs is a great command that reads streams of data from standard input, then generates and executes command lines; meaning it can take a command’s output and pass it as an argument for another command.
Some examples of the xargs command
type the command xargs and type something, once you are done press ctrl+d, it will echo back the string you typed.
$ xargs
hi
my name is jkoder.
now press ctrl+d
$ xargs
hi
my name is jkoder.
hi my name is jkoder.
$ echo a b c d e f| xargs
a b c d e f
If we want to spring 3 words per line then
$ echo a b c d e f| xargs -n 3
a b c
d e f
Say, if we want to delete files with extension .exe in a given folder, then we can use the below command also-
$ ls
abc.doc efg.exe klm.xml ijk.exe
$ find . -name "*.exe" | xargs rm -rf
$ ls
abc.doc klm.xml
So how will we use the xargs commands to execute the curl command in parallel? Suppose you want to get the HTTP headers of https://jkoder.com using the curl command. To do this, you would need to invoke the curl command with the -I flag followed by the URL as shown.
$ curl -I https://jkoder.com
HTTP/2 200
x-powered-by: PHP/7.0.33
content-type: text/html; charset=UTF-8
link: <https://jkoder.com/wp-json/>; rel="https://api.w.org/"
etag: "917-1655878567;;;"
x-litespeed-cache: hit
date: Wed, 22 Jun 2022 06:26:24 GMT
server: LiteSpeed
vary: User-Agent
content-security-policy: upgrade-insecure-requests;
alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"
Now let’s say we want to execute the above curl command in 5 parallel requests. For that we will use our curl command along with the xargs command, like the below one.
xargs -I % -P 5 curl -I "https://jkoder.com" < <(printf '%s\n' {1..10})
HTTP/2 200
HTTP/2 200
HTTP/2 200
x-powered-by: PHP/7.0.33
x-powered-by: PHP/7.0.33
x-powered-by: PHP/7.0.33
content-type: text/html; charset=UTF-8
content-type: text/html; charset=UTF-8
link: <https://jkoder.com/wp-json/>; rel="https://api.w.org/"
link: <https://jkoder.com/wp-json/>; rel="https://api.w.org/"
content-type: text/html; charset=UTF-8
etag: "917-1655878567;;;"
x-litespeed-cache: hit
etag: "917-1655878567;;;"
date: Wed, 22 Jun 2022 06:30:50 GMT
x-litespeed-cache: hit
server: LiteSpeed
link: <https://jkoder.com/wp-json/>; rel="https://api.w.org/"
date: Wed, 22 Jun 2022 06:30:50 GMT
vary: User-Agent
server: LiteSpeed
etag: "917-1655878567;;;"
content-security-policy: upgrade-insecure-requests;
vary: User-Agent
alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"
content-security-policy: upgrade-insecure-requests;
x-litespeed-cache: hit
alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"
date: Wed, 22 Jun 2022 06:30:50 GMT
server: LiteSpeed
vary: User-Agent
content-security-policy: upgrade-insecure-requests;
alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"
HTTP/2 200
x-powered-by: PHP/7.0.33
content-type: text/html; charset=UTF-8
link: <https://jkoder.com/wp-json/>; rel="https://api.w.org/"
etag: "917-1655878567;;;"
x-litespeed-cache: hit
date: Wed, 22 Jun 2022 06:30:50 GMT
server: LiteSpeed
vary: User-Agent
content-security-policy: upgrade-insecure-requests;
alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"
HTTP/2 200
x-powered-by: PHP/7.0.33
content-type: text/html; charset=UTF-8
link: <https://jkoder.com/wp-json/>; rel="https://api.w.org/"
etag: "917-1655878567;;;"
x-litespeed-cache: hit
date: Wed, 22 Jun 2022 06:30:52 GMT
server: LiteSpeed
vary: User-Agent
content-security-policy: upgrade-insecure-requests;
alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"
HTTP/2 200
x-powered-by: PHP/7.0.33
content-type: text/html; charset=UTF-8
link: <https://jkoder.com/wp-json/>; rel="https://api.w.org/"
etag: "917-1655878567;;;"
x-litespeed-cache: hit
date: Wed, 22 Jun 2022 06:30:52 GMT
server: LiteSpeed
vary: User-Agent
content-security-policy: upgrade-insecure-requests;
alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"
HTTP/2 200
x-powered-by: PHP/7.0.33
content-type: text/html; charset=UTF-8
link: <https://jkoder.com/wp-json/>; rel="https://api.w.org/"
etag: "917-1655878567;;;"
x-litespeed-cache: hit
date: Wed, 22 Jun 2022 06:30:52 GMT
server: LiteSpeed
vary: User-Agent
content-security-policy: upgrade-insecure-requests;
alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"
HTTP/2 200
x-powered-by: PHP/7.0.33
content-type: text/html; charset=UTF-8
link: <https://jkoder.com/wp-json/>; rel="https://api.w.org/"
etag: "917-1655878567;;;"
x-litespeed-cache: hit
date: Wed, 22 Jun 2022 06:30:52 GMT
server: LiteSpeed
vary: User-Agent
content-security-policy: upgrade-insecure-requests;
alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"
HTTP/2 200
x-powered-by: PHP/7.0.33
content-type: text/html; charset=UTF-8
link: <https://jkoder.com/wp-json/>; rel="https://api.w.org/"
etag: "917-1655878567;;;"
x-litespeed-cache: hit
date: Wed, 22 Jun 2022 06:30:52 GMT
server: LiteSpeed
vary: User-Agent
content-security-policy: upgrade-insecure-requests;
alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"
HTTP/2 200
x-powered-by: PHP/7.0.33
content-type: text/html; charset=UTF-8
link: <https://jkoder.com/wp-json/>; rel="https://api.w.org/"
etag: "917-1655878567;;;"
x-litespeed-cache: hit
date: Wed, 22 Jun 2022 06:30:53 GMT
server: LiteSpeed
vary: User-Agent
content-security-policy: upgrade-insecure-requests;
alt-svc: h3=":443"; ma=2592000, h3-29=":443"; ma=2592000, h3-Q050=":443"; ma=2592000, h3-Q046=":443"; ma=2592000, h3-Q043=":443"; ma=2592000, quic=":443"; ma=2592000; v="43,46"