Skip to content

Tips and Tricks

Jump to bottom Edit New page
Kelly Brazil edited this page Aug 22, 2022 · 15 revisions

Generic Converters

jc comes with a few generic or standard-format converters, including:

  • ASCII (and Unicode) Tables
  • Multi-line ASCII (and Unicode) Tables
  • CSV
  • INI
  • Key/Value
  • XML
  • YAML

Sometimes commands will output in one of these supported formats for easy conversion to JSON.

Example: CURL Headers

For example, you can parse the HTTP headers from curl using the Key/Value parser:

$ curl --silent --header "origin: https://origin.example.com" \
       --request GET "https://cors-test.appspot.com/test" \
       --head | jc --kv -p
{
  "http/2 200": "",
  "cache-control": "no-cache",
  "access-control-allow-origin": "https://origin.example.com",
  "access-control-max-age": "0",
  "access-control-allow-credentials": "true",
  "set-cookie": "test=test",
  "expires": "Fri, 01 Jan 1990 00:00:00 GMT",
  "content-type": "application/json",
  "x-cloud-trace-context": "086f2570a400102ed316c10f1719c87e",
  "date": "Tue, 17 May 2022 21:39:06 GMT",
  "server": "Google Frontend",
  "content-length": "15",
  "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""
}

Example: NMAP XML Output

Another example would be converting NMAP output to JSON. NMAP does not (yet) support native JSON output, but it does support XML output. You can use the jc --xml parser to get JSON output from NMAP:

$ nmap -sV -oX output.xml avleonov.com | jc --xml -p
{
  "nmaprun": {
    "@scanner": "nmap",
    "@args": "nmap -sV -oX output.xml avleonov.com",
    "@start": "1520760090",
    "@startstr": "Sun Mar 11 12:21:30 2018",
    "@version": "7.60",
    "@xmloutputversion": "1.04",
    "scaninfo": {
      "@type": "connect",
      "@protocol": "tcp",
      "@numservices": "1000",
      "@services": "1,3-4,6-7,9,13,17,19-26,30,32-33,37,42-43,49,53,70,79-85,88-90,99-100,106,109-111,113,119,125,135,139,143-144,146,161,163,179,199,211-212,222,254-256,259,264,280,301,306,311,340,366,389,406-407,416-417,425,427,443-445,458,464-465,481,497,500,512-515,524,541,543-545,548,554-555,563,587,593,616-617,625,631,636,646,648,666-668,683,687,691,700,705,711,714,720,722,726,749,765,777,783,787,800-801,808,843,873,880,888,898,900-903,911-912,981,987,990,992-993,995,999-1002,1007,1009-1011,1021-1100,1102,1104-1108,1110-1114,1117,1119,1121-1124,1126,1130-1132,1137-1138,1141,1145,1147-1149,1151-1152,1154,1163-1166,1169,1174-1175,1183,1185-1187,1192,1198-1199,1201,1213,1216-1218,1233-1234,1236,1244,1247-1248,1259,1271-1272,1277,1287,1296,1300-1301,1309-1311,1322,1328,1334,1352,1417,1433-1434,1443,1455,1461,1494,1500-1501,1503,1521,1524,1533,1556,1580,1583,1594,1600,1641,1658,1666,1687-1688,1700,1717-1721,1723,1755,1761,1782-1783,1801,1805,1812,1839-1840,1862-1864,1875,1900,1914,1935,1947,1971-1972,1974,1984,1998-2010,2013,2020-2022,2030,2033-2035,2038,2040-2043,2045-2049,2065,2068,2099-2100,2103,2105-2107,2111,2119,2121,2126,2135,2144,2160-2161,2170,2179,2190-2191,2196,2200,2222,2251,2260,2288,2301,2323,2366,2381-2383,2393-2394,2399,2401,2492,2500,2522,2525,2557,2601-2602,2604-2605,2607-2608,2638,2701-2702,2710,2717-2718,2725,2800,2809,2811,2869,2875,2909-2910,2920,2967-2968,2998,3000-3001,3003,3005-3007,3011,3013,3017,3030-3031,3052,3071,3077,3128,3168,3211,3221,3260-3261,3268-3269,3283,3300-3301,3306,3322-3325,3333,3351,3367,3369-3372,3389-3390,3404,3476,3493,3517,3527,3546,3551,3580,3659,3689-3690,3703,3737,3766,3784,3800-3801,3809,3814,3826-3828,3851,3869,3871,3878,3880,3889,3905,3914,3918,3920,3945,3971,3986,3995,3998,4000-4006,4045,4111,4125-4126,4129,4224,4242,4279,4321,4343,4443-4446,4449,4550,4567,4662,4848,4899-4900,4998,5000-5004,5009,5030,5033,5050-5051,5054,5060-5061,5080,5087,5100-5102,5120,5190,5200,5214,5221-5222,5225-5226,5269,5280,5298,5357,5405,5414,5431-5432,5440,5500,5510,5544,5550,5555,5560,5566,5631,5633,5666,5678-5679,5718,5730,5800-5802,5810-5811,5815,5822,5825,5850,5859,5862,5877,5900-5904,5906-5907,5910-5911,5915,5922,5925,5950,5952,5959-5963,5987-5989,5998-6007,6009,6025,6059,6100-6101,6106,6112,6123,6129,6156,6346,6389,6502,6510,6543,6547,6565-6567,6580,6646,6666-6669,6689,6692,6699,6779,6788-6789,6792,6839,6881,6901,6969,7000-7002,7004,7007,7019,7025,7070,7100,7103,7106,7200-7201,7402,7435,7443,7496,7512,7625,7627,7676,7741,7777-7778,7800,7911,7920-7921,7937-7938,7999-8002,8007-8011,8021-8022,8031,8042,8045,8080-8090,8093,8099-8100,8180-8181,8192-8194,8200,8222,8254,8290-8292,8300,8333,8383,8400,8402,8443,8500,8600,8649,8651-8652,8654,8701,8800,8873,8888,8899,8994,9000-9003,9009-9011,9040,9050,9071,9080-9081,9090-9091,9099-9103,9110-9111,9200,9207,9220,9290,9415,9418,9485,9500,9502-9503,9535,9575,9593-9595,9618,9666,9876-9878,9898,9900,9917,9929,9943-9944,9968,9998-10004,10009-10010,10012,10024-10025,10082,10180,10215,10243,10566,10616-10617,10621,10626,10628-10629,10778,11110-11111,11967,12000,12174,12265,12345,13456,13722,13782-13783,14000,14238,14441-14442,15000,15002-15004,15660,15742,16000-16001,16012,16016,16018,16080,16113,16992-16993,17877,17988,18040,18101,18988,19101,19283,19315,19350,19780,19801,19842,20000,20005,20031,20221-20222,20828,21571,22939,23502,24444,24800,25734-25735,26214,27000,27352-27353,27355-27356,27715,28201,30000,30718,30951,31038,31337,32768-32785,33354,33899,34571-34573,35500,38292,40193,40911,41511,42510,44176,44442-44443,44501,45100,48080,49152-49161,49163,49165,49167,49175-49176,49400,49999-50003,50006,50300,50389,50500,50636,50800,51103,51493,52673,52822,52848,52869,54045,54328,55055-55056,55555,55600,56737-56738,57294,57797,58080,60020,60443,61532,61900,62078,63331,64623,64680,65000,65129,65389"
    },
    "verbose": {
      "@level": "0"
    },
    "debugging": {
      "@level": "0"
    },
    "host": {
      "@starttime": "1520760091",
      "@endtime": "1520760120",
      "status": {
        "@state": "up",
        "@reason": "syn-ack",
        "@reason_ttl": "0"
      },
      "address": {
        "@addr": "81.177.6.213",
        "@addrtype": "ipv4"
      },
      "hostnames": {
        "hostname": [
          {
            "@name": "avleonov.com",
            "@type": "user"
          },
          {
            "@name": "srv36-h-st.jino.ru",
            "@type": "PTR"
          }
        ]
      },
      "ports": {
        "extraports": [
          {
            "@state": "filtered",
            "@count": "936",
            "extrareasons": [
              {
                "@reason": "no-responses",
                "@count": "916"
              },
              {
                "@reason": "host-unreaches",
                "@count": "20"
              }
            ]
          },
          {
            "@state": "closed",
            "@count": "56",
            "extrareasons": {
              "@reason": "conn-refused",
              "@count": "56"
            }
          }
        ],
        "port": [
          {
            "@protocol": "tcp",
            "@portid": "21",
            "state": {
              "@state": "open",
              "@reason": "syn-ack",
              "@reason_ttl": "0"
            },
            "service": {
              "@name": "ftp",
              "@product": "ProFTPD",
              "@method": "probed",
              "@conf": "10",
              "cpe": "cpe:/a:proftpd:proftpd"
            }
          },
          {
            "@protocol": "tcp",
            "@portid": "22",
            "state": {
              "@state": "open",
              "@reason": "syn-ack",
              "@reason_ttl": "0"
            },
            "service": {
              "@name": "ssh",
              "@product": "OpenSSH",
              "@version": "5.3",
              "@extrainfo": "protocol 2.0",
              "@method": "probed",
              "@conf": "10",
              "cpe": "cpe:/a:openbsd:openssh:5.3"
            }
          },
          {
            "@protocol": "tcp",
            "@portid": "80",
            "state": {
              "@state": "open",
              "@reason": "syn-ack",
              "@reason_ttl": "0"
            },
            "service": {
              "@name": "http",
              "@product": "nginx",
              "@method": "probed",
              "@conf": "10",
              "cpe": "cpe:/a:igor_sysoev:nginx"
            }
          },
          {
            "@protocol": "tcp",
            "@portid": "443",
            "state": {
              "@state": "open",
              "@reason": "syn-ack",
              "@reason_ttl": "0"
            },
            "service": {
              "@name": "http",
              "@product": "nginx",
              "@tunnel": "ssl",
              "@method": "probed",
              "@conf": "10",
              "cpe": "cpe:/a:igor_sysoev:nginx"
            }
          },
          {
            "@protocol": "tcp",
            "@portid": "2222",
            "state": {
              "@state": "open",
              "@reason": "syn-ack",
              "@reason_ttl": "0"
            },
            "service": {
              "@name": "ssh",
              "@product": "ProFTPD mod_sftp",
              "@version": "0.9.7",
              "@extrainfo": "protocol 2.0",
              "@method": "probed",
              "@conf": "10",
              "cpe": "cpe:/a:proftpd:proftpd:0.9.7"
            }
          },
          {
            "@protocol": "tcp",
            "@portid": "3306",
            "state": {
              "@state": "open",
              "@reason": "syn-ack",
              "@reason_ttl": "0"
            },
            "service": {
              "@name": "mysql",
              "@product": "MySQL",
              "@version": "5.5.59-MariaDB",
              "@method": "probed",
              "@conf": "10",
              "cpe": "cpe:/a:mysql:mysql:5.5.59-mariadb"
            }
          },
          {
            "@protocol": "tcp",
            "@portid": "8080",
            "state": {
              "@state": "open",
              "@reason": "syn-ack",
              "@reason_ttl": "0"
            },
            "service": {
              "@name": "http",
              "@product": "Apache httpd",
              "@method": "probed",
              "@conf": "10",
              "cpe": "cpe:/a:apache:http_server"
            }
          },
          {
            "@protocol": "tcp",
            "@portid": "8443",
            "state": {
              "@state": "open",
              "@reason": "syn-ack",
              "@reason_ttl": "0"
            },
            "service": {
              "@name": "ssl",
              "@product": "Apache httpd",
              "@extrainfo": "SSL-only mode",
              "@tunnel": "ssl",
              "@method": "probed",
              "@conf": "10",
              "cpe": "cpe:/a:apache:http_server"
            }
          }
        ]
      },
      "times": {
        "@srtt": "6937",
        "@rttvar": "1642",
        "@to": "100000"
      }
    },
    "runstats": {
      "finished": {
        "@time": "1520760120",
        "@timestr": "Sun Mar 11 12:22:00 2018",
        "@elapsed": "29.83",
        "@summary": "Nmap done at Sun Mar 11 12:22:00 2018; 1 IP address (1 host up) scanned in 29.83 seconds",
        "@exit": "success"
      },
      "hosts": {
        "@up": "1",
        "@down": "0",
        "@total": "1"
      }
    }
  }
}

Common String Conversions

jc comes with converters for common strings found in program output, files, etc. including:

  • URLs
  • IP Addresses
  • Email Addresses
  • ISO 8601 Datetimes
  • Unix Timestamps
  • JWT token strings
  • Syslog
  • CEF

Example: ISO-8601 Datetime

If your program output contains a standard ISO 8601 datetime string, you can explode all of the datetime parts to a JSON object:

$ date --iso-8601=seconds | jc --iso-datetime -p
{
  "year": 2022,
  "month": "Aug",
  "month_num": 8,
  "day": 1,
  "weekday": "Mon",
  "weekday_num": 1,
  "hour": 5,
  "hour_24": 17,
  "minute": 32,
  "second": 6,
  "microsecond": 0,
  "period": "PM",
  "utc_offset": "+0000",
  "day_of_year": 213,
  "week_of_year": 31,
  "iso": "2022-08-01T17:32:06+00:00",
  "timestamp": 1659375126
}

Alternative JSON Output

Most parsers have slightly different output when using the -r (Raw) cli option. The Raw output is typically used for troubleshooting or in cases where you don't want string to integer/float/bool conversions or even some calculated timestamp fields.

A couple parsers will output significantly different styles of JSON if the Raw option is used:

  • env
  • history

Example: env variations

For example, the env parser turns each environment variable into its own JSON object and adds all of these objects to a list:

$ env | jc --env -p
[
  {
    "name": "TERM_PROGRAM",
    "value": "Apple_Terminal"
  },
  {
    "name": "SHELL",
    "value": "/bin/bash"
  },
  {
    "name": "TERM",
    "value": "xterm-256color"
  },
  ...
]

This output makes it easy to iterate over each key/value pair and match values to predictable key names. But if you don't need that format, try the -r option:

$ env | jc --env -p -r
{
  "TERM_PROGRAM": "Apple_Terminal",
  "SHELL": "/bin/bash",
  "TERM": "xterm-256color",
  ...
}

With this output there is no list to iterate over - If you know the name of the key, you can quickly get the value.

Example: history variations

The history parser is similar:

$ history | jc --history -p
[
  {
    "line": 1009,
    "command": "jc git log | jq"
  },
  {
    "line": 1010,
    "command": "jc git log --format=full | jq"
  },
  {
    "line": 1011,
    "command": "jc git log --format=fuller --stat | jq"
  },
  {
    "line": 1012,
    "command": "jc git log --format=oneline --shortstat | jq"
  },
  ...
]

vs.

$ history | jc --history -p -r
{
  "1009": "jc git log | jq",
  "1010": "jc git log --format=full | jq",
  "1011": "jc git log --format=fuller --stat | jq",
  "1012": "jc git log --format=oneline --shortstat | jq",
  ...
}

© 2019-2023 Kelly Brazil

Add a custom sidebar
Clone this wiki locally