jq用法

jq用法

一、参数说明

  • r: 去掉结果中的 " 双引号
  • c: 在一行显示

二、函数说明

  • length: 统计键(值)总数
  • type: 获取元素类型
  • to_entries: 条目
  • map:数组中的元素进行映射转换
  • to_entries:
  • from_entries:
  • keys: 查找json中所有的键, 默认获取第一层键,指定键则获取子键
  • has:用来是判断是否存在某个key
  • sort_by: 数组排序
  • reduce: 累加

三、示例

3.1、显示所有值

格式化阅读json文件
1
  cat cfg.json | jq
直接打开文件
1
  jq . cfg.json
一行显示值
1
  jq -c . cfg.json

查找所有键

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
  $ jq 'keys' cfg.json
  [
	"conda",
	"function",
	"python",
	"source"
  ]

  $ jq '.source | keys' cfg.json
  [
	"url"
  ]
  $ jq '.source.url | keys' cfg.json
  [
	"acl-runtime",
	"ais_bench",
	"audio",
	"cann",
	"graphic",
	"miniconda",
	"mxvision",
	"wifi"
  ]

去掉方括号

1
2
3
4
5
6
7
8
9
  $ jq '.source.url | keys[]' cfg.json
  "acl-runtime"
  "ais_bench"
  "audio"
  "cann"
  "graphic"
  "miniconda"
  "mxvision"
  "wifi"

显示所有子值

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
  $ jq '.source[]' cfg.json
  {
	"miniconda": "https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-py39_23.1.0-1-Linux-aarch64.sh",
	"cann": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%207.0.RC1/Ascend-cann-toolkit_7.0.RC1_linux-aarch64.run",
	"mxvision": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/MindX/MindX%205.0.RC3/Ascend-mindxsdk-mxvision_5.0.RC3_linux-aarch64.run",
	"acl-runtime": "https://aisbench.obs.myhuaweicloud.com/packet/ais_bench_infer/0.0.2/aclruntime-0.0.2-cp39-cp39-linux_aarch64.whl",
	"ais_bench": "https://aisbench.obs.myhuaweicloud.com/packet/ais_bench_infer/0.0.2/ais_bench-0.0.2-py3-none-any.whl",
	"audio": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/audio.zip",
	"graphic": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/graphic.zip",
	"wifi": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/wifi.zip"
  }

  $ jq '.source.url[]' cfg.json
  "https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-py39_23.1.0-1-Linux-aarch64.sh"
  "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%207.0.RC1/Ascend-cann-toolkit_7.0.RC1_linux-aarch64.run"
  "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/MindX/MindX%205.0.RC3/Ascend-mindxsdk-mxvision_5.0.RC3_linux-aarch64.run"
  "https://aisbench.obs.myhuaweicloud.com/packet/ais_bench_infer/0.0.2/aclruntime-0.0.2-cp39-cp39-linux_aarch64.whl"
  "https://aisbench.obs.myhuaweicloud.com/packet/ais_bench_infer/0.0.2/ais_bench-0.0.2-py3-none-any.whl"
  "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/audio.zip"
  "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/graphic.zip"
  "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/wifi.zip"

去掉所有值 的双引号

1
2
3
4
5
6
7
8
9
  $ jq -r '.source.url[]' cfg.json
  https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-py39_23.1.0-1-Linux-aarch64.sh
  https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%207.0.RC1/Ascend-cann-toolkit_7.0.RC1_linux-aarch64.run
  https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/MindX/MindX%205.0.RC3/Ascend-mindxsdk-mxvision_5.0.RC3_linux-aarch64.run
  https://aisbench.obs.myhuaweicloud.com/packet/ais_bench_infer/0.0.2/aclruntime-0.0.2-cp39-cp39-linux_aarch64.whl
  https://aisbench.obs.myhuaweicloud.com/packet/ais_bench_infer/0.0.2/ais_bench-0.0.2-py3-none-any.whl
  https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/audio.zip
  https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/graphic.zip
  https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/wifi.zip

统计键值总数

1
2
3
4
5
  $ jq "length" cfg.json
  4

  $ jq ".source.url | length" cfg.json # 统计指定子键总数
  8

获取元素类型

1
2
  $ jq ".source.url | type" cfg.json
  "object"

取 [] 方括号中的值

1
2
  $ jq '.source.url["wifi"]' cfg.json
  "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/wifi.zip"

获取 json 中所有 key 和 value

1
2
3
4
5
6
7
  $ jq 'to_entries | map("\(.key)=\(.value | tostring)")' cfg.json
  [
	"source={\"url\":{\"miniconda\":\"https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-py39_23.1.0-1-Linux-aarch64.sh\",\"cann\":\"https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%207.0.RC1/Ascend-cann-toolkit_7.0.RC1_linux-aarch64.run\",\"mxvision\":\"https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/MindX/MindX%205.0.RC3/Ascend-mindxsdk-mxvision_5.0.RC3_linux-aarch64.run\",\"acl-runtime\":\"https://aisbench.obs.myhuaweicloud.com/packet/ais_bench_infer/0.0.2/aclruntime-0.0.2-cp39-cp39-linux_aarch64.whl\",\"ais_bench\":\"https://aisbench.obs.myhuaweicloud.com/packet/ais_bench_infer/0.0.2/ais_bench-0.0.2-py3-none-any.whl\",\"audio\":\"https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/audio.zip\",\"graphic\":\"https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/graphic.zip\",\"wifi\":\"https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/wifi.zip\"}}",
	"function={\"pre_process\":\"y\",\"apt_install\":\"y\",\"install_miniconda\":\"y\",\"python_pip_install\":\"y\",\"install_cann\":\"y\",\"install_mxvision\":\"y\",\"install_acllite\":\"y\",\"add_audio\":\"y\",\"add_local_desktop\":\"y\",\"add_remote_desktop\":\"y\",\"add_wifi\":\"y\",\"post_process\":\"y\"}",
	"conda={\"source\":\"channels:\\n - defaults\\n show_channel_urls: true\\n default_channels:\\n - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\\n - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r\\n - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2\\n custom_channels:\\n conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\\n msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\\n bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\\n menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\\n pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\\n pytorch-lts: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\\n simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\"}",
	"python={\"source\":\"[global]\\n index-url = https://pypi.tuna.tsinghua.edu.cn/simple\\n [install]\\n trusted-host = https://pypi.tuna.tsinghua.edu.cn\"}"
  ]

示例

key 和 value 的值组合

文件内容
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
$ jq to_entries cfg.json
[
  {
    "key": "source",
    "value": {
      "url": {
        "miniconda": "https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-py39_23.1.0-1-Linux-aarch64.sh",
        "cann": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%207.0.RC1/Ascend-cann-toolkit_7.0.RC1_linux-aarch64.run",
        "mxvision": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/MindX/MindX%205.0.RC3/Ascend-mindxsdk-mxvision_5.0.RC3_linux-aarch64.run",
        "acl-runtime": "https://aisbench.obs.myhuaweicloud.com/packet/ais_bench_infer/0.0.2/aclruntime-0.0.2-cp39-cp39-linux_aarch64.whl",
        "ais_bench": "https://aisbench.obs.myhuaweicloud.com/packet/ais_bench_infer/0.0.2/ais_bench-0.0.2-py3-none-any.whl",
        "audio": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/audio.zip",
        "graphic": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/graphic.zip",
        "wifi": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/wifi.zip"
      }
    }
  },
  {
    "key": "function",
    "value": {
      "pre_process": "y",
      "apt_install": "y",
      "install_miniconda": "y",
      "python_pip_install": "y",
      "install_cann": "y",
      "install_mxvision": "y",
      "install_acllite": "y",
      "add_audio": "y",
      "add_local_desktop": "y",
      "add_remote_desktop": "y",
      "add_wifi": "y",
      "post_process": "y"
    }
  },
  {
    "key": "conda",
    "value": {
      "source": "channels:\n - defaults\n show_channel_urls: true\n default_channels:\n - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r\n - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2\n custom_channels:\n conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n pytorch-lts: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud"
    }
  },
  {
    "key": "python",
    "value": {
      "source": "[global]\n index-url = https://pypi.tuna.tsinghua.edu.cn/simple\n [install]\n trusted-host = https://pypi.tuna.tsinghua.edu.cn"
    }
  }
]
用函数执行结果
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
 jq "to_entries | from_entries" cfg.json
{
  "source": {
    "url": {
      "miniconda": "https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-py39_23.1.0-1-Linux-aarch64.sh",
      "cann": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/CANN/CANN%207.0.RC1/Ascend-cann-toolkit_7.0.RC1_linux-aarch64.run",
      "mxvision": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/MindX/MindX%205.0.RC3/Ascend-mindxsdk-mxvision_5.0.RC3_linux-aarch64.run",
      "acl-runtime": "https://aisbench.obs.myhuaweicloud.com/packet/ais_bench_infer/0.0.2/aclruntime-0.0.2-cp39-cp39-linux_aarch64.whl",
      "ais_bench": "https://aisbench.obs.myhuaweicloud.com/packet/ais_bench_infer/0.0.2/ais_bench-0.0.2-py3-none-any.whl",
      "audio": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/audio.zip",
      "graphic": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/graphic.zip",
      "wifi": "https://ascend-repo.obs.cn-east-2.myhuaweicloud.com/Atlas%20200I%20DK%20A2/DevKit/images/resources/wifi.zip"
    }
  },
  "function": {
    "pre_process": "y",
    "apt_install": "y",
    "install_miniconda": "y",
    "python_pip_install": "y",
    "install_cann": "y",
    "install_mxvision": "y",
    "install_acllite": "y",
    "add_audio": "y",
    "add_local_desktop": "y",
    "add_remote_desktop": "y",
    "add_wifi": "y",
    "post_process": "y"
  },
  "conda": {
    "source": "channels:\n - defaults\n show_channel_urls: true\n default_channels:\n - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r\n - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2\n custom_channels:\n conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n pytorch-lts: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud\n simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud"
  },
  "python": {
    "source": "[global]\n index-url = https://pypi.tuna.tsinghua.edu.cn/simple\n [install]\n trusted-host = https://pypi.tuna.tsinghua.edu.cn"
  }
}

jq 处理 JSON 文件实战:数组合并与对象覆盖

背景

现有两个 JSON 文件:

  • package.json :包含 menus.commandPalette 数组。
  • add.json :包含 add_menu 数组。

目标:将 add_menu 的数组合并到 menus.commandPalette 中,并了解 jq 在数组、对象操作中的行为。

原始文件内容

通过以下命令创建两个示例文件:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# file1: package.json
cat - > package.json << 'EOF'
{
    "menus": {
        "commandPalette": [
            {
                "command": "go.test.refresh",
                "when": "false"
            }
        ]
    }
}
EOF

# file2: add.json
cat - > add.json << 'EOF'
{
    "add_menu": [
        {
            "command": "go.explorer.refresh",
            "when": "false"
        },
        {
            "command": "go.explorer.open",
            "when": "false"
        }
    ]
}
EOF

jq 核心操作分析

合并两个文件中的数组

使用 -s (–slurp)将多个输入合并为一个数组,然后通过索引访问并合并:

1
jq -s '.[0].menus.commandPalette = .[0].menus.commandPalette + .[1].add_menu | .[0]' package.json add.json
  • .[0] :第一个文件(package.json)
  • .[1] :第二个文件(add.json)
  • add_menu 数组追加到 commandPalette 后,输出修改后的第一个对象。

jq 对数组的操作

加法(+)用于数组合并:

1
echo '[1,2] [3,4]' | jq -s '.[0] + .[1]'
[1,2,3,4]

jq 对数字的操作

加法(+)用于数值相加:

1
echo '1 2' | jq -s '.[0] + .[1]'
3

jq 对对象的操作

对象合并时,相同 key 的值会被后面的对象覆盖:

1
echo '{"a":1, "b":2} {"b":21, "c":3}' | jq -s '.[0] + .[1]'
{
  "a": 1,
  "b": 21,
  "c": 3
}

演示:向数组中追加元素(不修改原文件)

使用 += 操作符向 add_menu 追加新对象:

1
jq '.add_menu += [{"data-root": "/data/disk/cache"}]' add.json
{
  "add_menu": [
    { "command": "go.explorer.refresh", "when": "false" },
    { "command": "go.explorer.open", "when": "false" },
    { "data-root": "/data/disk/cache" }
  ]
}

修改文件(安全方式)

使用临时文件避免数据丢失:

1
2
3
tmpfile=$(mktemp)
jq '.add_menu += [{"data-root": "/data/disk/cache"}]' add.json > "$tmpfile"
mv "$tmpfile" add.json

总结

  • jq -s 将多个输入合并为一个数组,便于跨文件操作。
  • + 操作符的行为取决于数据类型:

    • 数组:合并(拼接)
    • 数字:求和
    • 对象:合并(右侧覆盖左侧同名字段)
  • 使用 += 可方便地向数组或对象追加内容。
  • 修改文件时应先写入临时文件,再替换原文件,确保操作安全。
comments powered by Disqus
使用 Hugo 构建
主题 StackJimmy 设计