The Continent of International Law

Coding Instrument, Training, and Glossary

An elaborate coding instrument, with 10 sections and 500+ questions, is used to record the details of the international agreements. Section 1 records basic information about the agreement while Section 2 details the substance of the agreement, including the main prescriptions, proscriptions, and authorizations. Section 3 gathers information such as whether there are preambles, annexes, appendices, protocols, and references to other international agreements. Section 4 addresses issue scope while Section 5 details the membership provisions of the agreement, including the membership criteria, categories of membership, the number of states needed and/or the necessity of particular states to ratify the agreement before it enters into force, and the rights/responsibilities of non-state actors. Compliance provisions, including monitoring and dispute resolution, are recorded in Section 6, with questions about domestic implementing legislation and transition periods as well. Section 7 records the number of bodies the agreement creates, capturing details about membership, size, procedures for making decisions, and other details about the functioning and purpose of these bodies, in particular, delegated tasks. Section 8 delves into information exchange while the budgeting process is recorded in Section 9. Flexibility provisions constitute the substantive portion of Section 10 with questions about reviews, duration, extension, renegotiation, amendment, escape and withdrawal clauses, and termination procedures as well as information regarding reservations and declarations. Additionally, the characteristics of the parties (polity score, GDP, etc.) at the time the agreement was concluded are recorded using standard measures in the literature. For example, the literature includes measures that distinguish developed from developing states.

The reasonableness and consistency of the coding represent a very important issue in any data collection effort. Coder training is a 9-month process and includes two semester-long courses with the principal investigator (PI). Each coder must demonstrate a keen ability to translate qualitative evidence into quantitative assessments. Also, because standard practice suggests the use of multiple coders whose choices can be compared, every agreement in the dataset is coded independently by at least two coders. An intercoder reliability report is generated and highlights for each question for which there is a quantitative answer, like multiple choice, or yes/no, etc., disagreements among coders. In cases of disagreement, the responses are examined, and the disagreement resolved by the PI in consultation with the coders. The average coded agreement is characterized by disagreement on approximately 15 questions, or 4% of the quantitative questions; the range so far has been between 2% and 15%. Coding proceeds agreement by agreement. This approach (labeled AVFEC by Mitchell 2006:43) is used rather than a coding scheme whereby each variable across agreements is coded at the same time. While the AVFEC approach is criticized for reducing empirical regularity in the data coding, despite its efficiency in terms of understanding the detailed context of an agreement (which often helps a coder resolve the meaning of ambiguous provisions), this claim is alleviated for two reasons. First, as mentioned, intercoder reliability reports are generated, and, in the case of disagreement across coders, the disagreements are resolved by a group meeting that includes the PI (who remains the same over time). Second, before certain data are used for empirical testing, a third coder checks particular variables across all agreements. For example, as a check on the coding of precision, which unlike many variables requires a judgment call, a third coder looked at the entire set of agreements and focused only on precision. He therefore noticed any inconsistencies in coding given that he only had to keep one mental category in his head.

A glossary accompanies the coding instrument, with the main entries being the key terms (questions and answers) in the instrument. Definitions are based on Rational Design and other related theoretical work in this area and on the overall political science perspective on international law. In addition to definitions, there are examples of the various concepts drawing on some important international agreements with which the coders are trained. The glossary is available on the download page of this website.

This glossary is important for three reasons. First, to ensure intercoder-reliability, coders must be operating with the same, precise definitions. Even though they will learn these concepts as part of their training, many of these coders will be coding for two years. When confronted with many different kinds of agreements, they may want to double-check that their understanding is consistent with mine. Second, and surprisingly, some of these terms have never been succinctly defined. Concepts such as €œcompliance, delegation, and international body€ are the cornerstone of much international relations scholarship; yet many of these have never been carefully phrased with definitional boundaries. While many scholars may disagree with my definitions, making them public forces other scholars to be precise about how their conceptions differ. We in the international cooperation field are in need of such precision if we desire to advance both theoretically and empirically. (Precise definitions are particularly important for systematic comparative work.) Third, given the relationship developing between international law and international relations, comparison of their respective definitions will be quite useful for those crossing interdisciplinary boundaries.

Independent of the coding of the institutional design variables, students with training in rationalist approaches to international cooperation and I also look at the agreement. We answer the following substantive question among others: How can the cooperation problem be characterized? In addition to the independent variables elaborated in Rational Design, I have added the following possible answers to capture as fully as possible the range of problems states tend to face: domestic commitment/time inconsistency problem, encouraging positive externalities, discouraging negative externalities, breaking deadlock, and other. The other category captures areas of cooperation such as the exportation/codification of norms, pure coordination games, and foreign aid/development assistance. A detailed definition and example of each of these problems is given in the appendix. Of course, more than one answer can be chosen for each agreement. This gets around the problems of having to force real-life issues into a 2×2 game.

This part of the COIL project is significant because scholars have been talking about games of cooperation for decades. Nonetheless, measuring what these games might look like and their prevalence in real international agreements has heretofore not been attempted, despite their centrality in our theories. Obviously, these questions are not nearly as straightforward as those pertaining to agreement design. An inference must be made from the agreement to the cooperation problem. I see no way around this in such a study using a random sample of agreements given the observations are the agreements themselves and they cut across diverse issue and sub-issue areas. Nonetheless, there are some factors that I hope will alleviate concerns. First, the inference comes by looking at relevant background information. Sometimes, negotiators reveal the problems they were attempting to solve, and this is documented. Unfortunately, this is not always the case for an agreement from the random sample. Rather, research needs to be done more broadly on the particular states and on their relationships. For example, the political history of the states must be examined for the decade before the agreement is signed to determine whether the agreement may be serving to solve a domestic commitment problems – as Moravscik (2000) argues, newly democratized states often try to constrain future governments from violating human rights by signing human rights agreements. An another example, in a bilateral agreement, the relationship of the dyad in the decade or two before the agreement is signed is examined to discern whether one of the parties may have been uncertain about the preferences of the other. Research is also done on the general problems of the sub-issue at the time. Importantly, only the substantive goals of the agreement are looked at when trying to infer the underlying cooperation problem(s). Given that my theoretical work focuses on explaining the procedural or design aspects of the agreements, the separation of coders for, what are in my analyses, the independent and dependent variables is critical to the integrity of the project.