indexer-agent: Agent deploys subgraphs multiple times

[cause1ove]

When I add multiple offchain rules to indexer-agent, deployments are created multiple times and might be assigned to multiple nodes which leads to subgraphs failing. I'm running indexer-agent v0.20.3 with graph-node v0.27.0

Steps to reproduce:

• Run indexer-agent with multiple nodes configured (INDEXER_AGENT_INDEX_NODE_IDS=index_node_0,index_node_1,index_node_2,index_node_3)
• Add multiple offchain rules without delay

graph indexer rules offchain QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee
graph indexer rules offchain QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA
graph indexer rules offchain QmetcxTkUGpMWBN79HgvzYev31PNJqjB56qtnaKRQfurs7
graph indexer rules offchain QmRgcqzGHrnzPtd8BBvp18Lx184zf6yisxef6nV1EHjGkW
graph indexer rules offchain QmYvNqEnKw5hsExogDahfXdVapuz7HimiSoqhoiW5bpHJs
...

Result:

• indexer-agent tries to deploy the subgraph multiple times
• Different index nodes start syncing the same deployment (in my example, subgraph got reassigned 5 times to 3 different nodes)
• Indexing fails with there are most likely two (or more) nodes indexing this subgraph error

Logs:

# docker logs graph_indexer_agent -t | grep QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee | grep -A10 'Deployment changes' | grep -v 'Reconcile deployments'
2022-10-06T19:09:21.061374954Z {"level":30,"time":1665083361061,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","deploy":[{"bytes32":"0xd05877e0acdcc6e321787d584f0ffa2b695d1af57ef6f12ef5c4db68f816ed54","ipfsHash":"QmcMwkbXhthjpPn91nWNPaLZC9Vz5vCRwKRMTmoV7kqy3H"},{"bytes32":"0xd1c472224bfda4e0da75b1b2908e33ded5f20ba67b90dc576221af3a1f9d02ae","ipfsHash":"QmcTVf1FQmXByB5cBmLH2yQr4RexzSeLYTkHQjzhBzA24d"},{"bytes32":"0xe8a324e306c88a6f86e21e221505e6a089824d67cb1d041d2bf8da6ed9b99c67","ipfsHash":"QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA"},{"bytes32":"0xf5eb985c7d1ca6cb95cd2e266ac1501f50bb252d5d3dbc4f5827ce8cdc65231e","ipfsHash":"QmetcxTkUGpMWBN79HgvzYev31PNJqjB56qtnaKRQfurs7"},{"bytes32":"0xf6febc5fb0c75ed811d4b5803cf7fd34e7fb8854f571e036015871176d095681","ipfsHash":"QmexpHwQQ2SkB4HVsjLCHs7RgzdQGjnsKAT99dJfpy2QUY"},{"bytes32":"0x0da25e1ac8a87c5d631b1680c3def679782b1489fe64454b0259c1c5725a02b3","ipfsHash":"QmPFscEgtdQ1xgnjhQdazw4GAC2Q1ENaLWDC7NyayuJHnS"},{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},{"bytes32":"0x31b091b737cb2f5abf965db8835a76026392014731aac77def6c01b8d73bc8a7","ipfsHash":"QmRgcqzGHrnzPtd8BBvp18Lx184zf6yisxef6nV1EHjGkW"},{"bytes32":"0x67236dab258a02ff9d9b9a919092ffeb847cbae1d3f4edc6b1d0cb9ee9e02b1d","ipfsHash":"QmVHG31Xon47RwD898JRzCuTZQFxuKxzKR99MXMse89wdE"},{"bytes32":"0x9d38aa1fce50de3ce5233a5b45c809cf8c9c98c5bad1ce2f4a2691d7f2fed804","ipfsHash":"QmYvNqEnKw5hsExogDahfXdVapuz7HimiSoqhoiW5bpHJs"},{"bytes32":"0xa7dda6a37a55be5979ddc4878da232bd8c4b3717014c5b0a2696d50daa79c579","ipfsHash":"QmZdvpm6yMmHcBJtYKLK5TzW4VVC3cmuJW6w6sQnkKu2wz"}],"remove":[],"msg":"Deployment changes"}
2022-10-06T19:09:21.063686345Z {"level":30,"time":1665083361063,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Index subgraph deployment"}
2022-10-06T19:09:21.179530392Z {"level":30,"time":1665083361179,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","deploy":[{"bytes32":"0xd1c472224bfda4e0da75b1b2908e33ded5f20ba67b90dc576221af3a1f9d02ae","ipfsHash":"QmcTVf1FQmXByB5cBmLH2yQr4RexzSeLYTkHQjzhBzA24d"},{"bytes32":"0xe8a324e306c88a6f86e21e221505e6a089824d67cb1d041d2bf8da6ed9b99c67","ipfsHash":"QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA"},{"bytes32":"0xf5eb985c7d1ca6cb95cd2e266ac1501f50bb252d5d3dbc4f5827ce8cdc65231e","ipfsHash":"QmetcxTkUGpMWBN79HgvzYev31PNJqjB56qtnaKRQfurs7"},{"bytes32":"0xf6febc5fb0c75ed811d4b5803cf7fd34e7fb8854f571e036015871176d095681","ipfsHash":"QmexpHwQQ2SkB4HVsjLCHs7RgzdQGjnsKAT99dJfpy2QUY"},{"bytes32":"0x0da25e1ac8a87c5d631b1680c3def679782b1489fe64454b0259c1c5725a02b3","ipfsHash":"QmPFscEgtdQ1xgnjhQdazw4GAC2Q1ENaLWDC7NyayuJHnS"},{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},{"bytes32":"0x31b091b737cb2f5abf965db8835a76026392014731aac77def6c01b8d73bc8a7","ipfsHash":"QmRgcqzGHrnzPtd8BBvp18Lx184zf6yisxef6nV1EHjGkW"},{"bytes32":"0x67236dab258a02ff9d9b9a919092ffeb847cbae1d3f4edc6b1d0cb9ee9e02b1d","ipfsHash":"QmVHG31Xon47RwD898JRzCuTZQFxuKxzKR99MXMse89wdE"},{"bytes32":"0x9d38aa1fce50de3ce5233a5b45c809cf8c9c98c5bad1ce2f4a2691d7f2fed804","ipfsHash":"QmYvNqEnKw5hsExogDahfXdVapuz7HimiSoqhoiW5bpHJs"},{"bytes32":"0xa7dda6a37a55be5979ddc4878da232bd8c4b3717014c5b0a2696d50daa79c579","ipfsHash":"QmZdvpm6yMmHcBJtYKLK5TzW4VVC3cmuJW6w6sQnkKu2wz"}],"remove":[],"msg":"Deployment changes"}
2022-10-06T19:09:21.181528197Z {"level":30,"time":1665083361181,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Index subgraph deployment"}
2022-10-06T19:09:21.318661553Z {"level":30,"time":1665083361318,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Deploy subgraph deployment"}
2022-10-06T19:09:21.394692627Z {"level":30,"time":1665083361394,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Deploy subgraph deployment"}
2022-10-06T19:09:22.954917883Z {"level":30,"time":1665083362954,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Successfully deployed subgraph deployment"}
2022-10-06T19:09:22.954921833Z {"level":30,"time":1665083362954,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"node":"index_node_0","msg":"Reassign subgraph deployment"}
2022-10-06T19:09:23.155471475Z {"level":30,"time":1665083363155,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Successfully deployed subgraph deployment"}
2022-10-06T19:09:23.155475504Z {"level":30,"time":1665083363155,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"node":"index_node_1","msg":"Reassign subgraph deployment"}
2022-10-06T19:09:40.876018564Z {"level":30,"time":1665083380875,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","deploy":[{"bytes32":"0xd1c472224bfda4e0da75b1b2908e33ded5f20ba67b90dc576221af3a1f9d02ae","ipfsHash":"QmcTVf1FQmXByB5cBmLH2yQr4RexzSeLYTkHQjzhBzA24d"},{"bytes32":"0xe8a324e306c88a6f86e21e221505e6a089824d67cb1d041d2bf8da6ed9b99c67","ipfsHash":"QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA"},{"bytes32":"0xf5eb985c7d1ca6cb95cd2e266ac1501f50bb252d5d3dbc4f5827ce8cdc65231e","ipfsHash":"QmetcxTkUGpMWBN79HgvzYev31PNJqjB56qtnaKRQfurs7"},{"bytes32":"0xf6febc5fb0c75ed811d4b5803cf7fd34e7fb8854f571e036015871176d095681","ipfsHash":"QmexpHwQQ2SkB4HVsjLCHs7RgzdQGjnsKAT99dJfpy2QUY"},{"bytes32":"0x0da25e1ac8a87c5d631b1680c3def679782b1489fe64454b0259c1c5725a02b3","ipfsHash":"QmPFscEgtdQ1xgnjhQdazw4GAC2Q1ENaLWDC7NyayuJHnS"},{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},{"bytes32":"0x31b091b737cb2f5abf965db8835a76026392014731aac77def6c01b8d73bc8a7","ipfsHash":"QmRgcqzGHrnzPtd8BBvp18Lx184zf6yisxef6nV1EHjGkW"},{"bytes32":"0x67236dab258a02ff9d9b9a919092ffeb847cbae1d3f4edc6b1d0cb9ee9e02b1d","ipfsHash":"QmVHG31Xon47RwD898JRzCuTZQFxuKxzKR99MXMse89wdE"},{"bytes32":"0x9d38aa1fce50de3ce5233a5b45c809cf8c9c98c5bad1ce2f4a2691d7f2fed804","ipfsHash":"QmYvNqEnKw5hsExogDahfXdVapuz7HimiSoqhoiW5bpHJs"},{"bytes32":"0xa7dda6a37a55be5979ddc4878da232bd8c4b3717014c5b0a2696d50daa79c579","ipfsHash":"QmZdvpm6yMmHcBJtYKLK5TzW4VVC3cmuJW6w6sQnkKu2wz"}],"remove":[],"msg":"Deployment changes"}
2022-10-06T19:09:40.877586602Z {"level":30,"time":1665083380877,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Index subgraph deployment"}
2022-10-06T19:09:41.026132078Z {"level":30,"time":1665083381025,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","deploy":[{"bytes32":"0xc7c272666799ff602fd8fae8ab986f1b052f8873891ef29364aca09286ab08fa","ipfsHash":"QmbnRp17QCAKsmS7js8n99F2UZTQkXWQ6ePi6ekdL5f45P"},{"bytes32":"0xd1c472224bfda4e0da75b1b2908e33ded5f20ba67b90dc576221af3a1f9d02ae","ipfsHash":"QmcTVf1FQmXByB5cBmLH2yQr4RexzSeLYTkHQjzhBzA24d"},{"bytes32":"0xe8a324e306c88a6f86e21e221505e6a089824d67cb1d041d2bf8da6ed9b99c67","ipfsHash":"QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA"},{"bytes32":"0xf5eb985c7d1ca6cb95cd2e266ac1501f50bb252d5d3dbc4f5827ce8cdc65231e","ipfsHash":"QmetcxTkUGpMWBN79HgvzYev31PNJqjB56qtnaKRQfurs7"},{"bytes32":"0xf6febc5fb0c75ed811d4b5803cf7fd34e7fb8854f571e036015871176d095681","ipfsHash":"QmexpHwQQ2SkB4HVsjLCHs7RgzdQGjnsKAT99dJfpy2QUY"},{"bytes32":"0x0da25e1ac8a87c5d631b1680c3def679782b1489fe64454b0259c1c5725a02b3","ipfsHash":"QmPFscEgtdQ1xgnjhQdazw4GAC2Q1ENaLWDC7NyayuJHnS"},{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},{"bytes32":"0x31b091b737cb2f5abf965db8835a76026392014731aac77def6c01b8d73bc8a7","ipfsHash":"QmRgcqzGHrnzPtd8BBvp18Lx184zf6yisxef6nV1EHjGkW"},{"bytes32":"0x67236dab258a02ff9d9b9a919092ffeb847cbae1d3f4edc6b1d0cb9ee9e02b1d","ipfsHash":"QmVHG31Xon47RwD898JRzCuTZQFxuKxzKR99MXMse89wdE"},{"bytes32":"0x9d38aa1fce50de3ce5233a5b45c809cf8c9c98c5bad1ce2f4a2691d7f2fed804","ipfsHash":"QmYvNqEnKw5hsExogDahfXdVapuz7HimiSoqhoiW5bpHJs"},{"bytes32":"0xa7dda6a37a55be5979ddc4878da232bd8c4b3717014c5b0a2696d50daa79c579","ipfsHash":"QmZdvpm6yMmHcBJtYKLK5TzW4VVC3cmuJW6w6sQnkKu2wz"}],"remove":[],"msg":"Deployment changes"}
2022-10-06T19:09:41.027515047Z {"level":30,"time":1665083381027,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Index subgraph deployment"}
2022-10-06T19:09:41.154043888Z {"level":30,"time":1665083381153,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Deploy subgraph deployment"}
2022-10-06T19:09:41.171100073Z {"level":30,"time":1665083381170,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Deploy subgraph deployment"}
2022-10-06T19:09:42.501661296Z {"level":30,"time":1665083382501,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Successfully deployed subgraph deployment"}
2022-10-06T19:09:42.501666386Z {"level":30,"time":1665083382501,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"node":"index_node_2","msg":"Reassign subgraph deployment"}
2022-10-06T19:09:42.518375413Z {"level":30,"time":1665083382518,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Successfully deployed subgraph deployment"}
2022-10-06T19:09:42.518378643Z {"level":30,"time":1665083382518,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"node":"index_node_2","msg":"Reassign subgraph deployment"}
2022-10-06T19:10:20.027238150Z {"level":30,"time":1665083420027,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","deploy":[{"bytes32":"0xc7c272666799ff602fd8fae8ab986f1b052f8873891ef29364aca09286ab08fa","ipfsHash":"QmbnRp17QCAKsmS7js8n99F2UZTQkXWQ6ePi6ekdL5f45P"},{"bytes32":"0xd1c472224bfda4e0da75b1b2908e33ded5f20ba67b90dc576221af3a1f9d02ae","ipfsHash":"QmcTVf1FQmXByB5cBmLH2yQr4RexzSeLYTkHQjzhBzA24d"},{"bytes32":"0xe8a324e306c88a6f86e21e221505e6a089824d67cb1d041d2bf8da6ed9b99c67","ipfsHash":"QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA"},{"bytes32":"0xf5eb985c7d1ca6cb95cd2e266ac1501f50bb252d5d3dbc4f5827ce8cdc65231e","ipfsHash":"QmetcxTkUGpMWBN79HgvzYev31PNJqjB56qtnaKRQfurs7"},{"bytes32":"0xf6febc5fb0c75ed811d4b5803cf7fd34e7fb8854f571e036015871176d095681","ipfsHash":"QmexpHwQQ2SkB4HVsjLCHs7RgzdQGjnsKAT99dJfpy2QUY"},{"bytes32":"0x0da25e1ac8a87c5d631b1680c3def679782b1489fe64454b0259c1c5725a02b3","ipfsHash":"QmPFscEgtdQ1xgnjhQdazw4GAC2Q1ENaLWDC7NyayuJHnS"},{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},{"bytes32":"0x31b091b737cb2f5abf965db8835a76026392014731aac77def6c01b8d73bc8a7","ipfsHash":"QmRgcqzGHrnzPtd8BBvp18Lx184zf6yisxef6nV1EHjGkW"},{"bytes32":"0x67236dab258a02ff9d9b9a919092ffeb847cbae1d3f4edc6b1d0cb9ee9e02b1d","ipfsHash":"QmVHG31Xon47RwD898JRzCuTZQFxuKxzKR99MXMse89wdE"},{"bytes32":"0x9d38aa1fce50de3ce5233a5b45c809cf8c9c98c5bad1ce2f4a2691d7f2fed804","ipfsHash":"QmYvNqEnKw5hsExogDahfXdVapuz7HimiSoqhoiW5bpHJs"},{"bytes32":"0xa7dda6a37a55be5979ddc4878da232bd8c4b3717014c5b0a2696d50daa79c579","ipfsHash":"QmZdvpm6yMmHcBJtYKLK5TzW4VVC3cmuJW6w6sQnkKu2wz"}],"remove":[],"msg":"Deployment changes"}
2022-10-06T19:10:20.029254924Z {"level":30,"time":1665083420029,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","component":"Agent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Index subgraph deployment"}
2022-10-06T19:10:20.178769502Z {"level":30,"time":1665083420178,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Deploy subgraph deployment"}
2022-10-06T19:10:21.040350122Z {"level":30,"time":1665083421040,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/JUukdeJsee","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"msg":"Successfully deployed subgraph deployment"}
2022-10-06T19:10:21.040472301Z {"level":30,"time":1665083421040,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","deployment":{"bytes32":"0x1bd330d2a8d041b6538ef60f8cf6327bc9ba520bdd104ea4e15fcc3a31cf1fc7","ipfsHash":"QmQDGVhAjCh3gkdMznBMUkSUhPZEK5mwdjYqJUukdeJsee"},"node":"index_node_2","msg":"Reassign subgraph deployment"}

[cause1ove]

In the logs I've also found that every reconcile there is an error with one of the subgraphs, so agent tries to retry an entire queue which leads to problems.

Full logs: https://pastebin.com/QK9C3TEH

2022-10-06T19:09:21.350344021Z {"level":50,"time":1665083361350,"pid":1,"hostname":"7f5d923e16e4","name":"IndexerAgent","name":"indexer-agent/p5hqPvjYeA","deployment":{"bytes32":"0xe8a324e306c88a6f86e21e221505e6a089824d67cb1d041d2bf8da6ed9b99c67","ipfsHash":"QmdzmaeKuBroxvAdRrENGKyCAMvAresZuKXop5hqPvjYeA"},"err":{"type":"IndexerError","message":"Failed to deploy subgraph deployment","stack":"IndexerError: Failed to deploy subgraph deployment\n    at indexerError (/opt/indexer/packages/indexer-common/dist/errors.js:161:12)\n    at Indexer.<anonymous> (/opt/indexer/packages/indexer-agent/dist/indexer.js:694:63)\n    at Generator.next (<anonymous>)\n    at fulfilled (/opt/indexer/packages/indexer-agent/dist/indexer.js:5:58)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)","code":"IE026","explanation":"https://github.com/graphprotocol/indexer/blob/master/docs/errors.md#ie026","cause":{"code":0,"message":"subgraph validation error: [the graft base is invalid: deployment not found: QmQCH1zo9JPFhqEJSHUziWu3TtDFHWWAB4bNdBwhm2ua14]"}},"msg":"Failed to deploy subgraph deployment"}

[fordN]

Hey @cause1ove, thanks for the report. We have a few related updates in the works that should resolve the issues you experienced.

• For the situation where the agent repeatedly adds actions to the queue that can be expected to fail (have failed recently).
• Additionally we are working on an update to allow the graph-nodes to decide which index node a deployment is assigned to, so the agent will no longer try to specify or reassign deployments.

[cause1ove]

@fordN just to clarify: in my case the subgraph was deploying successfully every time but was redeployed at every reconcile. As far as I understood it was because the other subgraph was failing to deploy. Will these fixes cover these cases?

[fordN]

Resolved by #868