Defence and military data and the IISS

The International Institute for Strategic Studies (IISS) celebrated its 60th anniversary in 2018. Initially called the Institute of Strategic Studies (the word ‘International’ was adopted in the early 1970s), the then ISS was created in 1958, enabled by an initial three-year grant from the Ford Foundation, to consider issues including defence and military strategy after the advent of nuclear weapons and to provide objective information on the military balance of power in the context of Cold War confrontation. The IISS was not conceived as a think tank that would primarily send papers with policy recommendations to governments, but rather as an organization that would help to create the basis for assessments that could underlie decision making by providing data and analysis (see Howard, 2020) independent of government or political affiliation.¹ Its ethos since then has been focused more on helping analysts and practitioners with the question of how to think about a problem rather than telling them what to think.

Within the context of this edited volume, the IISS forms part of the epistemic infrastructure in the area of defence and military matters. It has instituted a range of processes and publications that consciously aim to produce data and knowledge about the phenomena encountered in this field of enquiry and policy. However, the methodologies employed by the IISS in its defence assessment work should not be considered as an abstract evolution of comparative methodologies. They form part of comparative practices that are informed by, and ultimately reflect judgements on, the evolution of wider political landscapes, and thus relevance criteria with regard to what should be counted and compared, and how this should be accomplished.

One of the major products the IISS produces to achieve this is The Military Balance, published for the first time in 1959 and annually since. At the time it was a thin 11-page document containing information on the defence holdings and policies of 15 states, collated essentially by the founding director of the IISS, Alasdair Buchan. Today the book covers 173 states and territories, is some 500 pages strong every year, and is created by a growing team of professional defence analysts based in several IISS offices and organized in its Defence and Military Analysis Programme (DMAP). The underpinning general idea, that better data could (not would) lead to better decisions and that a reliable independent-of-government reference source might help to avoid misunderstandings and increase transparency, remain essentially the same. With reference to the project that gave rise to this edited volume, The Military Balance helps to map important elements of military capability across units of analysis – in this case nation states – and across time.

The 1959 edition noted in its foreword that the publication was intended as ‘a contribution to the growing concern that is developing throughout the world about the arms race’ – there would be value in bringing together available information ‘into one simple comparative analysis … in order to provide a firmer basis, not only of the discussion of “the balance of terror”, but of the problems of disarmament’ (ISS, 1959: Foreword). The 1961 foreword suggests ‘the demand for previous editions … has shown that [The Military Balance] fills an important gap in public knowledge, and is considered useful as a guide to the strategic balance between the great powers and their allies and of the orders of magnitude involved in the problem of disarmament’ (ISS, 1961: 1). Later versions, such as the one published in 1990 (IISS, 1990), carried a disclaimer that was actually titled ‘WARNING’ in capital letters. This advisory note, written to help readers interpret what they were seeing, explained that the data in The Military Balance provided a quantitative assessment of personnel strengths and equipment inventories, but not an assessment of military capability. It went on to suggest that ‘those who wish to do so can use the data provided to construct their own force comparisons’ (IISS, 1990: 10). Indeed, it was recognized within the Institute that the very title of the book reflected not only this distinction between quantitative and qualitative assessments, but also the challenges that a product like The Military Balance could create for analysts. According to Sir Michael Howard, one of the founders of the Institute, Alasdair Buchan and his successors were ‘later to lament that they had got themselves stuck with the title The Military Balance, providing as it does so stark and conceptually misleading an idea of the complex nature of military power’. But, he continued, ‘stuck they are, and “MilBal” has become the Institute’s flagship’ (Howard, 2020: 287).

Unlike the first editions, the 2022 edition does not explicitly explain why the IISS produces the book and collects data but instead provides extensive notes on how this is done and how the data and analysis contained in it should be interpreted (see IISS, 2022a). The modern editions all clarify that The Military Balance is an open-source work, based on either the most accurate data available or on the best estimate that can be made and that, while the cooperation of governments has been sought and in many cases received, the Institute’s judgements contained in The Military Balance are its own: ‘The data presented reflects judgements based on information available to the IISS at the time the book is compiled’ (IISS, 2022a: 511) – a statement contained in the methodology statements in the book and database and that has changed little in recent years.

Since 2017, the IISS has published the Military Balance Plus, a searchable electronic database that contains the dataset and text and graphics content contained in the Military Balance books, as well as additional data the IISS possesses but cannot include in the book, and additional pieces of analysis including some from related IISS research (see IISS, 2023b).

The database is updated continuously throughout the year, and this now represents the latest assessment available from the IISS at any given point in time, whereas the book continues to be published annually. While the database removes many constraints on the data the IISS can provide – including volume and pace of update – an electronic database that users can interrogate according to their needs will likely also change patterns and practices of interaction with the dataset provided. In very simple terms, extracting a tailored response covering multiple data years, multiple countries and multiple capability categories – for example – can now be achieved in seconds rather than the hours, or possibly even days, that would be required to manually extract the same information from multiple volumes of the book. If users know they can test hypotheses or assumptions much more rapidly, they will achieve results faster but also are likely to ask more questions, enabled by the gain in productivity. Users are also able to export data or results of a complex query into preformatted Excel sheets which facilitate integration of this dataset with other sources or existing datasets held by the user. Because the database introduces functionality and datasets not present in the print book, it opens to subscribers the chance to interrogate datasets in complex ways and juxtapose data to produce insightful conclusions (one example is to produce dissimilar datasets, such as combat aircraft against air defence systems in potential adversary countries). At the same time, subscribers are able, in calls with IISS account managers, to suggest to the IISS new functionalities, data sets and user journeys (linking up data sets according to their interests) that they would find useful and that go beyond what is included in the database. Within resource and technological constraints, DMAP considers this feedback in future development and update cycles. As a result, the database has created a feedback loop between the creators of the data and those using it that is much deeper and wider than before.

Drawing on the work conducted in the context of The Military Balance publication and database, this chapter will examine a range of defence economic and military capability metrics that have been implemented by the IISS DMAP in order to enable the practice of international military comparisons. Specifically, it will present different metrics to measure defence expenditure and discuss their advantages and effects, illustrate changing naval equipment classifications over time, and present a methodology to classify military equipment by degree of modernity. These examples can serve to illustrate the challenges but also the evolving nature of the practice of comparison in the field The Military Balance covers. The chapter will conclude with a section analysing the drivers that influence the way the IISS produces military data and assessments, or, in the language of the present volume, comparative knowledge.

Show me the money: defence spending

On the face of it, the question of how much a government spends on defence each year should be straightforward to answer. However, internationally there is no shared understanding of what constitutes a defence budget. Some countries do not even release a defence budget publicly, others classify parts or all of it, and yet others use line items in other budgets to supplement defence activity. So, for international defence spending comparisons, the first problem is what to include.² NATO includes in its official definition of defence expenditure the defence budget, pensions, costs of peacekeeping and humanitarian operations, and R&D costs. Where possible, the IISS seeks to follow this approach internationally. Military Balance products list three different measures of basic defence spending data: the official defence budget figure provided by the government (where it exists), an additional measure referred to as defence expenditure where additional outlays for defence not covered in the official defence budget are known to exist or can be estimated and, for NATO countries, a defence expenditure figure as reported by NATO (converting local currencies using International Monetary Fund (IMF) exchange rates). Of these, the second defence expenditure figure is thus an estimate that is relevant where the IISS assesses the official budget to be an incomplete picture of the total financial effort devoted to defence. Therefore, the defence expenditure figure can be expected to be higher than the official budget for the same year.

There are additional challenges beyond definitions and transparency, such as the sector-specific impact of inflation on defence spending or the utility and feasibility of applying purchasing power parity (PPP) to defence – both aspects that have wide ranging implications for the interpretation of the data and what kind of output can be generated (see McGerty, 2022). Hence a seemingly simple question like ‘how much did Russia spend on defence in 2021?’ has in fact multiple answers (figures taken from IISS, 2022a):

1. a.The official Russian defence budget at market exchange rates: US$45.8 billion
2. b.Defence expenditure at market exchange rates: US$62.2 billion
3. c.Defence expenditure in purchasing power parity: US$178 billion

This matters for policy discourse, for example on threat perceptions and regional security dynamics. European policy makers regularly ask how much of a conventional military threat Russia really can be to EU and NATO member states if its defence budget is less than US$50 billion. However, if examined via a PPP measure the story is somewhat different, with Russian spending in the above 2021 example close to the combined expenditure of France, Germany and the UK.

The Military Balance uses market exchange rates based on IMF data. The limitation of this approach is that it does not consider the cost basis – personnel, equipment and investment – that might differ quite dramatically between countries. Converting figures from local currency to US dollars at market rates masks the fact that these input costs will be lower in some countries. An alternative approach would be to make conversions using PPP exchange rates. However, the appropriateness of PPP conversions depends on the extent to which a country is self-sufficient in developing and producing the armaments required by its armed forces.

PPP conversions have utility in reaching a more nuanced understanding of the defence expenditure of China and Russia, as imported systems play almost no role in Russia’s case and only a small and decreasing one in China’s. As a consequence, the IISS began to include a PPP-based expenditure figure for both countries several years ago. However, PPP conversions are less suitable when assessing the spending of countries such as India and Saudi Arabia, which rely heavily on imports of military equipment from relatively high-cost producers. For those countries it would be necessary to adopt a hybrid approach to determine defence expenditure in dollars, with the market exchange rate used for converting defence procurement and the PPP conversion rate applied to all other defence expenditure (personnel, operations and so on). So, to produce standardized international comparisons, PPP conversions would have to be applied to all countries (IISS, 2023a: 491).

Moreover, using PPP exchange rates creates its own analytical challenges, not least because defence-specific PPP rates are unlikely to be the same as the general GDP-based PPP rates widely used in economics. The latter reflect mostly civilian goods and services, whereas a measure more relevant to defence spending would need to take into account defence input unit costs which are difficult to establish principally because of transparency issues. Creating defence-specific PPP measures is thus a very resource-intensive undertaking, which so far has prevented their widespread use beyond a few particularly relevant countries.

Inflation presents a similar challenge to the PPP discussion. The Military Balance provides, as part of its national economic statistics included in the country entries, an inflation data point which represents the year-on-year change in consumer prices in that country. However, it is reasonable to assume that defence inflation, and thus an inflation figure tailored to defence activity, would be different from consumer price changes because it would cover very different goods. The UK Ministry of Defence (2022) has found that defence inflation has been significantly higher than inflation faced by the consumers in the general population: 4.1 per cent in the 2020–21 period compared to 0.6 per cent (since then consumer inflation has risen to a 40-year high in the UK, peaking at the end of 2022). It has furthermore found that 36 per cent of defence inflation is related to labour-cost inflation but 63 per cent is driven by contract-related inflation (UK Ministry of Defence, 2022). While efforts exist to assess defence-specific inflation, there is so far no shared methodology and approach that would enable international comparison.

A second set of problems concerns what metrics to use when making comparisons: absolute spending in local currency or in USD (or some other currency)? Defence spending as a percentage of GDP or a percentage of overall government spending? These metrics generate very different results. In absolute terms the US ranks number one in the world, but in terms of defence spending as a percentage of GDP, it was 18th in 2022, with Oman being at the top of the board. Of course, it makes a difference what data sources are used for GDP data. For instance, the World Bank, Organisation for Economic Cooperation and Development (OECD) and European Commission all present differing figures. This makes it important also to examine, in turn, the methodologies used by these data providers. Indeed, as the US State Department’s World Military Expenditure and Arms Transfers publications have shown, there are several different methodologies to covert expenditure from local currencies into US dollars, all with somewhat different outcomes.³

The Military Balance Plus database provides additional datapoints with relevance for defence economics, most notably budget breakdown data and defence budget forecasts. The budget breakdown data provide totals in local and USD terms (current and constant) as well as a percentage of defence investments as part of the entire defence budget in order to indicate how much of a defence budget is devoted to defence research and development and weapons procurement. This, in turn, can be used as an input into assessment of defence modernization processes and equipment recapitalization across countries. In July 2020, the IISS launched a forecasting tool in the database to determine future defence budget trends for some 30 countries covering some 90 per cent of global defence spending. The IISS defence budget forecast uses a Bayesian network-based model. The IISS determined the most important factors that shape defence budget allocations and, together with a data-driven model based on historical data, developed an econometric forecast model. The algorithm was developed in partnership with data scientists at Objective Computing Limited, a UK-based developer. Up to 45 variables are considered for each country model. Input variables include macroeconomic factors (gross domestic product, government expenditure), demographics (dependency ratio), defence economic data (corruption, arms trade, military aid, defence-industrial complex, official spending targets), regime type (democratic, military rule), and war and international relations dynamics (years of warfare, alliance membership, missions and deployments). Certain variables are logged based on plot evaluation and analyst discussion of variable behaviour. These forecasts can be used to identify plausible defence budget trends which can, for example, be compared to defence policy and procurement ambitions.

Classifying and categorizing equipment, and moving to qualitative judgements

A key goal the IISS pursues with The Military Balance publication, both in database and book form, is to provide an authoritative reference point for military and defence data. It allows examination of national forces and equipment, and also international comparisons across groups of countries. It is used by a diverse audience ranging from government officials, the armed forces, the private sector, media and members of the analytical community; all these audiences have somewhat different requirements and interests. The Military Balance includes data on state armed forces equipment holdings both for the active fleet (the available inventory) and, where possible, equipment held in storage (even though equipment held in store is not counted in the main inventory totals published by the IISS). IISS data is necessarily selective. It does not cover small arms and light weapons. These are of course central to many conflicts, but their ubiquity makes any assessment of numbers and organization a highly fluid and complex endeavour. Because of this, IISS data encompass crew-served weapons and above. The same analytical challenge is presented by other munitions such as missiles and bombs, but in these cases the IISS assesses numbers of missile launchers, and types of air-launched missiles and bombs.

Technological evolution and the changing character of armed conflict also influence the roles in which military platforms are employed. These roles can, in turn, affect equipment classifications. For the IISS, it is not of overriding importance how a certain country classifies its equipment – or what name it gives it – but where a platform fits into the IISS equipment classification. The IISS classification system is designed to enable international comparisons, based on a defined set of criteria. As technology and the character of conflict evolve, these parameters might need to be revised, creating possible reclassification issues.

Classifying naval vessels according to role – the function they perform– is a complex undertaking. A post-war consensus on primary surface combatants revolved around a distinction between independently operating cruisers, air defence escorts (destroyers) and anti-submarine-warfare escorts (frigates). However, ships are increasingly performing a range of roles. Also, modern ship design has meant that the full-load displacement (FLD) of different warship types has evolved and, in some cases overlaps, further eroding what once were relatively clear distinctions. For these reasons, The Military Balance now classifies vessels by an assessed combination of role, equipment fit and displacement.

Principal surface combatants are multi-mission combat ships capable of complex warfighting and open ocean and task group operations, with an FLD above 2,200 tonnes. Cruisers sit above 9,750 tonnes FLD. Destroyers range from 4,500 to 9,749 tonnes FLD and will have principally a medium or an area air defence equipment fit and role, designed primarily for task group operations. They will generally be carrying a heavier armament than frigates. Frigates can range from 2,200 to 9,000 tonnes FLD and will have principally an anti-submarine warfare or general-purpose equipment fit and role. In the IISS classification, principal aviation-capable combat vessels are included among principal surface combatants. Full-size aircraft carriers have above 35,000 tonnes FLD, are conventionally or nuclear powered and capable of simultaneously mounting offensive and defensive operations with a fixed-wing aircraft group (conventional or short take-off and vertical landing (STOVL)).

This means that designations used by countries, for instance for their frigates or destroyers, in many cases match IISS designations, but sometimes they do not. For example, Japan’s Hyuga and Izumo ship-classes – listed by the Japan Maritime Self-Defence Force as helicopter-carrying destroyers – are classified by the IISS as helicopter carriers, reflective of their assessed role. As the conversion of the Izumo class to embark the F-35B combat aircraft approaches completion, the IISS will again reclassify the Izumo class as the vessels’ role changes. In 2020–21 the maritime data team working on Military Balance data undertook to reassess and, where necessary, reclassify data on the above parameters. This led to some movement between IISS maritime data classifications, as seen in in Table 3.1.

However, these assessments still lead to primarily quantitative outcomes. But many users across government, the armed forces, academia and the private sector have an analytical interest not just in understanding numbers of platforms and the roles they can perform, but also to get a measure of the capability of one type of platform versus another within a given platform classification. This requirement has led to a significant change in the type of defence data that The Military Balance provides. IISS data teams still produce hard facts around numbers and types of organizations and equipment, but now add a qualitative assessment of capability.

Of course, a host of input measures, including policy, doctrine, funding, maintenance and training, influence military capability. Nonetheless, judgements can be made which provide important indicators of capability. For example, there is a vast difference between a modern main battle tank and a main battle tank that is obsolete in capability terms. Indeed, while NATO member states’ inventories include more than 9,000 main battle tanks, in 2021 the IISS assessed some 40 per cent of them as either ageing or obsolete.

The IISS has conducted work to add assessments of equipment capability to its data set, adding new data (in the form of an analytical judgement made by DMAP staff) and delivering new functionality to users of the Military Balance Plus database in 2022. Subscribers can now query elements of the dataset based on the level of modernity of platforms. Through this step the IISS now provides data indicating the relative level of quality and capability of the equipment in the database. The IISS is incrementally increasing these judgements across its dataset, to cover a larger volume of data. Subscribers can view these judgements on the Military Balance Plus dashboard alongside an appropriate equipment record. For instance, the display for the Greek army’s main battle tanks indicates that the Leopard 2A6HEL is judged as ‘modern (+)’, the Leopard 2A4 as ‘modern’ while the Leopard 1A5s are ‘Ageing (+)’ and the army’s M48A5 Pattons are obsolescent. Aside from providing qualitative capability assessments that add a new dimension to force comparisons between countries, doing this also allows the IISS to better track the modernization of equipment inventories, in terms of both a current snapshot and a trendline over time. While primarily a feature of the database, the resulting output will also be included in graphics and charts in the Military Balance book.

The solution designed by DMAP aims to assign a numbered capability category to all equipment records in particular equipment classifications. For DMAP analysts engaged in assigning judgements on the internal Military Balance content management system, the value ‘1’ represents obsolete equipment and ‘5’ represents advanced equipment. This assignment needs to be based on technical characteristics specific to each equipment classification, as what makes a main battle tank obsolete or advanced naturally differs from what makes a fighter-ground attack aircraft obsolete or advanced. The capability judgement based on these specific characteristics generates a value for the numbered category which in turn translates into a label such as ‘obsolete’ or ‘advanced’ on the subscriber-facing Military Balance Plus dashboard. So, while the basic framework of making platform capability judgements needs to be applied across the domains (such as air, land, sea) and use identical language, the relationship between the characteristics observed and the capability judgement reached needs to be unique in order to yield useable information.

While the capability categories will remain fixed, the characteristics that lead to a particular capability judgement will evolve over time: what might be modern now will likely be obsolete at some point in the future. It is also necessary to capture upgrades and interim capabilities that lead to some types of equipment falling between capability categories, possessing some but not all of the technical characteristics required for a particular category. In a database format, where upgrades move equipment in the direction of a higher capability category, these instances can be displayed through symbols like plus signs (‘+’). Since this approach has been implemented, Military Balance Plus users have new functionality available that allows them to search classifications such as ‘main battle tanks’ for judgements of equipment capability. These judgements are available in the Military Balance Plus data tools module (where users can select multiple countries and equipment classifications for their searches) and are also displayed in the relevant ‘country and organization’ and ‘equipment’ modules. The search returns a breakdown of capability categories for one or more countries against the selected classifications, both in table form and as a colour-coded stacked bar chart.

The five categories the IISS is using for this judgement are ‘obsolete’ (1), ‘obsolescent’ (2), ‘ageing’ (3), ‘modern’ (4) and ‘advanced’ (5). It is important to underline that these judgements are based on a platform’s technical characteristics, not its physical age, remaining service life or the ability of the country that owns it to operate it successfully. Within these, IISS analysts have detailed lists to aid decision making, for instance as to which precise characteristics mean that the levels of protection for a set of equipment lead it to be assigned a particular category, be it obsolete, advanced or something in between (see Table 3.2).

Factors driving change in military data and defence assessments

The evolution of the IISS’s military and defence data and assessments process reflects a blend of the Institute’s original mission, availability of data, available resources, technological developments and the requirements of the three core audiences in government, the private sector, and the expert and opinion-forming communities. If the starting point in the late 1950s was to create one simple comparison, the journey since has been one of increasing complexity. More countries are covered now, more data areas are included and in greater depth and detail, more written analysis is provided and, whereas the first The Military Balance was essentially produced by a team of one, at the end of 2022, DMAP consisted of more than 20 full-time defence analysts. But the aim is still to provide the best possible open-source assessment based on the available information in order to inform the public policy debate.

New military technologies have emerged and a number of technologies that were developed in the civilian realm turned out to have military applications, either intended and unintended. Some, like cyber capabilities, have turned into their own domains of military competition and conflict. Armed forces have set up cyber commands and units at various levels to conduct defensive and offensive cyber operations. The rise of this domain and the military capability of nation states to operate in it needs to be captured in The Military Balance as well. The IISS started out with prose assessments of key developments in cyber strategy, doctrine and organization at the national level. It has since built a methodology for assessing, principally relating to indicators of militarily owned cyber capability and will in the coming years integrate the resulting new metrics into the dataset. Both The Military Balance 2021 and The Military Balance 2022 carried short chapters outlining the emerging thinking on this domain at the IISS (IISS, 2021a: 503–6; IISS, 2022a: 507–10). Another research team at the IISS, focused on cyber power and future conflict, is conducting wider assessments of national cyber power, beyond the military realm, that informs and contributes to the efforts within DMAP (see for example IISS, 2021b). A principal challenge for DMAP analysts is that capability in this area is opaque; equipment cannot be assessed as in other domains and so indicators of cyber capability are the focus of research attention. Moreover, the organizations pursuing activity in this area operate across the boundary between civil and military organizations. Signals intelligence agencies, for instance, are part of the relevant ‘organization’ list for countries as much as those organizations formally within the armed forces; in the cyber realm these civil and military organizations are both essential to the delivery of the capability.

Another important area where the boundaries are increasingly blurred relates to space systems. These have traditionally been used by armed forces and defence establishments for early warning, surveillance, and communications. And the cost of accessing space, and indeed of building satellite systems, traditionally restricted the ‘user club’. No more. A growing number of countries, and private sector firms, offer access to space at increasingly competitive costs, also offering bandwidth on commercial satellites. This complicates the analytical task, and in future means that when displaying space systems IISS analysis will have to take greater account of non-government and non-military providers; these are increasingly ‘militarily relevant’.

Other examples of technological change have triggered notable modifications within existing domains. The emergence, integration and fielding of uninhabited systems, which in its early years centred on the air domain, has long reached into the maritime and land domains as well. This trend generates a new information requirement and the need to create new classification elements capturing the different systems in use by governments. For example, the Uninhabited Aerial Vehicle (UAV) parent classification the IISS uses breaks down further into Combat, Intelligence, Surveillance, and Reconnaissance (CISR); Intelligence, Surveillance, and Reconnaissance (ISR); Electronic Warfare (EW); and Transport (TPT). Some of these are then further subdivided – for example, into heavy, medium and light – to form the full classification tree. Similar efforts are underway for the maritime and land domain. The IISS, to provide another example, also decided to add a ‘Loitering & Direct Attack Munitions’ classification to break out a category of weapons systems that in public discourse were sometimes grouped together with UAVs but have distinct characteristics. The IISS defined these new systems as air vehicles with an integral warhead that share some characteristics with both UAVs and cruise missiles. They are designed to fly either directly to their target (Direct Attack), or into a search or holding pattern (Loitering).

Another factor that is technology-driven is the ability of analysts to process greater amounts of data and utilize sources of data that would not have been available in the open-source environment until recently. Data scientists work with defence analysts to automate the creation of certain data sets for further assessment, reducing the data collection burden on analysts and creating greater time for interpretation, evaluation and assessment. High-resolution satellite imagery is now nearly ubiquitous, and a number of commercial providers have struck agreements with think tanks and media outlets to trade access to imagery for visibility of their brand (through the time-honoured technique of requesting acknowledgement of the source). The fact that commercial operators can make resources available that would have been reserved to a small number of governments around the world until a few years ago has levelled the playing field to a degree (Strobel and Wall, 2022). It has contributed to an ever-increasing volume of data and assessments offered by open-source analysts. These, in turn, have reinforced government efforts to integrate and exploit open-source material for their own efforts, in many cases mixing it with classified sources. This proliferation of methods, techniques and judgements is naturally of varied quality and sifting through the material to discern what is background noise and what is valuable insight has in itself become a challenge. Even so, government analysts continue to have access to confidential insights provided by diplomats and other government officials with relevant access, as well as classified material acquired by their intelligence agencies, sometimes from highly technical sources (IISS, 2022b). IISS analysts, working in the non-government open-source arena, are able to take advantage of the profusion of open-source data and the increasing range of technical tools to help gather and filter this information. But this change in the ‘data landscape’ makes more important associated moves within the Military Balance team to continue to improve the underlying analytical standards, techniques and processes that underpin its judgements.

That said, the reality of conflict – not least Russia’s war of aggression against Ukraine – demonstrates that an abundance of information does not necessarily lead to sound judgement. To simplify greatly, the performance of the Russian armed forces in the first year of the war was worse than external observers anticipated, and that of the armed forces of Ukraine was better (Dalsjö et al, 2022). An important caveat is, of course, that the Russian ground forces were by Moscow’s own metrics the least ‘modern’ of its armed forces and that some elements of the Russian armed forces have been used sparingly (at time of writing). Another caveat is that the assessments were in general not wrong in terms of the existence of equipment or weapons systems but in Russia’s ability to deliver the latent capability offered by this equipment. Moreover, the war in Ukraine underlines the importance of qualitative factors in capability assessments (Giegerich and Hackett, 2022). These include human factors such as the will to fight, morale, cohesion, and the quality of leadership, but also plans and training as well as defence-industrial resilience and logistics strategies. Among the many challenges highlighted by these qualitative factors is that while comparative assessments among states, an important function for the IISS The Military Balance, are difficult to achieve, their importance cannot be ignored. This also highlights that while instruments and institutions such as The Military Balance provide an important and elaborate epistemic infrastructure that informs practices of comparison, the latter can never be entirely reduced to the former. On the contrary, while improvements to methodologies are designed to provide as neutral and sound a basis for military force comparisons as possible, the requirement to compare factors that cannot easily be quantified as well requires constant reflection on both the analytical possibilities as well as the limitations of comparisons such as those in The Military Balance.