the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 21 Oct 2024
InsNet-CRAFTY v1.0: Integrating institutional network dynamics powered by large language models with land use change simulation
Abstract. Understanding and modelling environmental policy interventions can contribute to sustainable land use and management but is challenging because of the complex interactions among various decision-making actors. Key challenges include endowing modelled actors with autonomy, accurately representing their relational network structures, and managing the often-unstructured information exchange. Large language models (LLMs) offer new ways to address these challenges through the development of agents that are capable of mimicking reasoning, reflection, planning, and action. We present InsNet-CRAFTY (Institutional Network – Competition for Resources between Agent Functional Types) v1.0, a multi-LLM-agent model with a polycentric institutional framework coupled with an agent-based land system model. The numerical experiments simulate two competing policy priorities: increasing meat production versus expanding protected areas for nature conservation. The model includes a high-level policy-making institution, two lobbyist organisations, two operational institutions, and two advisory agents. Our findings indicate that while the high-level institution tends to avoid extreme budget imbalances and adopts incremental policy goals for the operational institutions, it leaves a budget deficit in one institution and a surplus in another unresolved. This is due to the competing influence of multiple stakeholders, which leads to the emergence of a path-dependent decision-making approach. Despite errors in information and behaviours by the LLM agents, the network maintains overall behavioural believability, demonstrating error tolerance. The results point to both the capabilities and challenges of using LLM agents to simulate policy decision-making processes of bounded rational human actors and complex institutional dynamics, such as LLM agents’ high flexibility and autonomy, alongside the complicatedness of agent workflow design and reliability in coupling with existing programmed land use systems. These insights contribute to advancing land system modelling and the broader field of institutional analysis, providing new tools and methodologies for researchers and policy-makers.
- Preprint
(5455 KB) - Metadata XML
- BibTeX
- EndNote
Status: open (until 16 Dec 2024)
-
RC1: 'Comment on egusphere-2024-2661', Anonymous Referee #1, 05 Dec 2024
reply
Summary of the review
Zeng et al. developed an innovative LLM model that simulates interactions between institutional agents that can mimic reasoning, planning and action. The model is novel because it addresses key challenges that learning and memory and polycentricity and because it is linked to an agent-based model that simulates changes in land use and livelihoods. The development of the LLM model described in the paper is ambitious and challenging. Certainly, it cannot be expected that all issues and challenges are yet addressed and that it completely functions as intended. The authors describe the challenges they occur and how they may solve them. It is impressive that the authors make sure that everything becomes available open access.
I read the paper with pleasure. It is generally well-written, novel and informative. However, there are a number of things that need improving in my opinion. That is why I recommend major revisions. In the attached pdf file, the authors can find detailed comments. Below they can find a summary of the issues that are in, my opinion, important to address.
The experimental set up:
The intention of the paper is to test the model and simulate institutional actor’s behavior in the land system. Many different types of policy goals can be tested and different types of actors with different types of profiles and ways in which they interact can be chosen. The choices made in the experimental set up affect the outcome of the experiment. At the moment, limited rationale is provided for the experimental set up by the authors. There is no rationale provided for the choice of the SSP-RCP scenario. Additionally, limited rationale is provided for the choice of starting conditions and the types of policies that are considered. Limited rationale is provided for the choice of the combination of agents in the experiment. As this all influences the outcomes of the experiment, it is important that such rationales are provided. There is limited rationale provided for focusing only on the response of institutional agents to EU land system dynamics, without considering effects other regions in the world may have had on the results. Additionally, it is important to discuss how the experimental set up could have influenced the outcomes in unintended ways and which limitations of the model could have been accidentally missed because they did not come into play because of the way the experiment was set up. It would be great if the authors could address this thoroughly in the paper, so that the value of doing this particular experiment, but also its limitations, becomes clearer. At the moment, it was difficult for me to judge if the model is sufficiently tested using through this one experiment to run other types of scenarios with other policy targets, other institutional agents etc. Or whether more tests and sensitivity analyses are necessary for the model to be used more broadly. Especially since the outcomes of budget surplus for PAs and budget deficit for agriculture are a bit counter-intuitive in my perception and seem to reveal an overreliance of agents on policy documents.
Errors and robustness:
The authors speak of error proneness, error tolerance and robustness but these terms are not defined and the process of testing for this is not explained in the methodology. Usually, these terms are used in modelling literature in the context of quantitative sensitivity analysis but here they are used to refer to some unexpected or undesirable behavior of institutional agents. I find this personally a bit confusing, as I do not see so well how the error and robustness of the model could be derived from a qualitative assessment of the agents’ behavioral patterns. Therefore, I would recommend to either use different terms, such as undesired or unexpected agent behavior or to really well define the terms around error well and thoroughly describe in the methodology how the authors assessed the errors. If the authors would really like to emphasize error proneness and robustness in the more traditional modelling sense, I would recommend the authors to do additional analyses. For example, to add a sensitivity analysis with different starting conditions, different environmental and social goals or different combinations of agents with different profiles, etc.
Information lacking to interpret results well:
As the LLM model is linked to CRAFTY, it is of course not possible to describe every dynamic of both models in detail in this paper. However, to understand the results and discussion some fundamental modelling assumptions were missing from the main paper, such as the way the budget is modelled and how the agents, for example know how much budget is needed etc. It would be great if the authors could provide more detailed descriptions of such assumptions and modelling choices, so that the results can be more easily interpreted. Or to very explicitly refer the appendices that are adjusted in such a way that the reader can understand the results and their interpretation easily after reading them.
Writing:
Although the model is intended to mimic real-life situations, the description of the model and the results, as well as the discussion of the results remains very high-level and abstract. I would highly recommend including real-life examples of agents in the context of the EU and to discuss the results in context of dynamics at play in the EU. This would all make it a bit more tangible. In particular I would recommend including a discussion of the outcome of the model in context of what happened in the EU in the past and what has been found in previous studies.
Writing style:
The paper is generally well-written. Yet, in some parts of the paper jargon is used and quite some terms that would be up for interpretation remain undefined. It would be good to more specifically define some of the terms, so that the model and results are easier to interpret by readers of different disciplines. This is important because the model can be used in interdisciplinary settings and, when linked to other models, such as CRAFTY can influence land use modelling, which is a different field again altogether. I have put comments throughout the paper that are hopefully helpful to address this.
Data sets
InsNet-CRAFTY v1.0 [data set] Yongchao Zeng, Calum Brown, Mohamed Byari, Joanna Raymond, Thomas Schmitt, and Mark Rounsevell https://doi.org/10.5281/zenodo.13944650
Model code and software
InsNet-CRAFTY v1.0 [code] Yongchao Zeng, Calum Brown, Mohamed Byari, Joanna Raymond, Thomas Schmitt, and Mark Rounsevell https://doi.org/10.5281/zenodo.13356487
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 139 | 34 | 12 | 185 | 4 | 5 |
- HTML: 139
- PDF: 34
- XML: 12
- Total: 185
- BibTeX: 4
- EndNote: 5
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1