By David Vohradsky, CGEIT, CRISC. First published in the ISACA Journal, Volume 2, 2015, as the lead feature of that issue, and republished here in full with permission. The controls, ratings and test plans throughout are a worked example, not a measured assessment of any organisation.

One of the responsibilities of line management in many organisations (particularly in financial services) is to provide assurance to the chief executive officer (CEO) and executives that high-rated risk factors are managed and that appropriate controls are in place and operating effectively. With increases in the regulatory regime, increasing technology complexity and pressures on cost, organisations are seeking productivity improvements in the evaluation of the performance of internal controls. One method of productivity improvement is applying technology to allow near continuous (or at least high-frequency) monitoring of control operating effectiveness, known as continuous controls monitoring (CCM). CCM is a subset of continuous assurance, alongside continuous data assurance (verifying the integrity of data flowing through systems) and continuous risk monitoring and assessment (dynamically measuring risk).

Improved management and monitoring of controls through CCM (and associated risk management activities) may reduce the extent to which audit and assurance staff need to undertake annual detailed testing of controls. In addition to cost reductions through improved efficiency and effectiveness (figure 1), other benefits include increased test coverage (through greater sampling and the ability to do more with the same or less labour), improved timeliness of testing, reduced risk velocity and potentially reduced remediation cost, greater visibility (when included in a governance, risk and compliance [GRC] solution), improved consistency, and the ability to identify trends. CCM also allows the replacement of manual, error-prone preventive controls with automated detective controls in which this would reduce the risk profile.

Figure 1: Indicative cost-benefit of CCM — cost plotted against rigour for CSA, SOC Type I reports, detailed testing, SOC Type II reports and CCM in its first, second and third years, with CCM falling in cost each year while rigour stays high — **Figure 1** — Indicative cost-benefit of CCM.

The steps for implementing CCM include:

Identify potential processes or controls according to industry frameworks such as COSO, COBIT 5 and ITIL; define the scope of control assurance based on business and IT risk assessments; and establish priority controls for continuous monitoring.
Identify the control objectives (or goals) and key assurance assertions for each control objective. (Guidelines for the formalisation of assertions may need to be developed as the concept of formal assertions is not well developed within IT risk).
Define a series of automated tests (or metrics) that will highlight (or suggest) success or failure of each assertion using a “reasonable person holistic review.”
Determine the process frequencies in order to conduct the tests at a point in time close to when the transactions or processes occur.
Create processes for managing the generated alarms, including communicating and investigating any failed assertions and ultimately correcting the control weakness.

Defining Controls to Monitor

The scope of overall IT control assurance is usually determined from critical business and IT processes, which are prioritised based on risk and prior experience in reviewing the controls through audits, self-assessments and control breakdowns. For the purposes of example, one can assume the organisation has determined a scope of annual control assurance based on the controls in figure 2.

Of these controls, the priorities for implementation of CCM should be based on risk ratings/return on investment (ROI) (such as value to the organisation) and ease of implementation (such as having readily available data from systems and controls that already have an aspect of monitoring and reporting).

Figure 2: Priority of controls for continuous monitoring. Cells marked as a candidate were highlighted green in the original; cells marked as a roadblock were highlighted red.
In Scope Controls	System	Monitored	Metrics	Risk*	Audit ROI†
Vendor service level agreement (SLA) management	Partial	No	No?		Medium
Software development life cycle (SDLC)	No	No	Yes		Low
Human resources (HR) management	Partial	Operational	No		Medium
User access reviews	Partial (roadblock)	No (roadblock)	Partial	High	High
Segregation of duties	No	No	No (roadblock)		High
Change management (candidate)	Yes (candidate)	Operational (candidate)	Yes (candidate)	High (candidate)	High (candidate)
Incident management	Yes	Operational	Yes		Low
Backup and recovery	Yes	Operational	Yes		Low
Capacity, availability and performance	Partial	Operational	Yes		Medium
IT service continuity	Partial	No	Yes		Medium
IT perimeter security	Yes	Alerts?	No? (roadblock)	High	High
AV management (candidate)	Yes (candidate)	Alerts (candidate)	Yes (candidate)	Medium	High (candidate)
Data loss prevention (candidate)	Yes (candidate)	Alerts (candidate)	Yes (candidate)	High (candidate)	High (candidate)
End point encryption	Yes	Alerts	Yes		Medium
Security monitoring	Yes	Operational	Yes	Medium (roadblock)	High

Figure 2 — Priority of controls for continuous monitoring. * Self-assessed. † Focus areas. Cells marked as a candidate were highlighted green in the original, and cells marked as a roadblock were highlighted red.

In the figure 2 example, the high-profile controls highlighted by the internal audit function have been assessed against data availability and existing monitoring or metrics. Controls highlighted in green are candidates for continuous control monitoring (red indicates a roadblock that may preclude a control from being considered). The priority or suitability of controls for continuous monitoring also needs to consider the relationships among controls. For example, configuration and vulnerability management rely on asset management, which may be deficient and not suitable for inclusion in the scope of assurance. In such a case, the controls that depend on it may not be suitable for continuous monitoring.

Identifying Assertions

Processes for management assurance of controls are usually more informal than an audit because they are often based on professional judgment, rather than detailed testing. An audit is a systematic process in which a qualified team or person objectively obtains and evaluates evidence regarding assertions about a process and forms an opinion on the degree to which the assertion is implemented. To automate an assurance process, control descriptions need to be reviewed to separate those components of the control that can be formally tested and those components that will rely on professional judgement.

Internal control objectives in a business context are categorised against five assertions used in the COSO model — existence/occurrence/validity, completeness, rights and obligations, valuation, and presentation and disclosure. These assertions have been expanded in the SAS 106, “Audit Evidence,” and, for the purposes of a technology context, can be restated in generic terms, as shown in figure 3.

Figure 3: SAS 106 financial statement assertions.
Classification	Assertion
Assertions about classes of transactions and events	Occurrence: Transactions and events that have been recorded have occurred.
	Completeness: All transactions and events that should have been recorded have been recorded.
	Accuracy: Data related to the transactions and events have been recorded appropriately.
	Cut-off: Transactions and events have been recorded in the correct period.
	Classification: Transactions and events have been recorded in proper accounts.
Assertions about account balances (assets)	Existence: The assets exist.
	Rights and obligations: The entity holds or controls the rights to assets.
	Completeness: All assets that should have been recorded have been recorded.
	Valuation and allocation: Assets are included in financial statements.
Assertions about presentation and disclosure	Occurrence, rights and obligations: Disclosed transactions and events have occurred.
	Completeness: All disclosures that should have been included have been included.
	Classification and understanding: Information is appropriately presented and described, and disclosures are clearly expressed.
	Accuracy and valuation: Financial and other information are disclosed fairly and at appropriate amounts.

Figure 3 — SAS 106 financial statement assertions.

COSO objectives are known as enterprise goals, IT-related goals and enabler goals in COBIT 5, and the financial statement assertions are loosely translated in the technology context to “completeness, accuracy, validity and restricted access.” Much (if not all) of the literature on CCM relates to business processes, and, as such, there is no documented alignment or mapping among IT control objectives (or goals) and the formal assertions necessary for formalised objective testing.

In an attempt to bridge this gap, figure 4 compares example control descriptions against related guidance from an IT security context and the related COBIT 5 goals, and proposes a formal assertion that could be used in a CCM context.

Figure 4: Proposed formal assertions for selected controls.
Example Control Description	ISO 27002 Guidance	COBIT 5 Process Goals	Proposed Formal Assertions
All changes to the IT systems (including hardware, networks and software) are managed to minimise the likelihood of disruption, unauthorised alterations and errors.	12.5.1 (e) Obtaining formal approval for detailed proposals before work commences 12.5.1 (f) Ensuring authorized users accept changes prior to implementation 12.5.1 (i) Maintaining an audit trail of all change requests	BAI06: (a) Authorised changes are made in a timely manner and with minimal errors. (b) Impact assessments reveal the effect of the change on all affected components. (c) All emergency changes are reviewed and authorised after the change. (d) Key stakeholders are kept informed of all aspects of the change.	CM1 An authorisation has occurred prior to every change. CM2 Testing has been completed for all changes prior to implementation. CM3 An approval has occurred, indicating completeness of testing conducted. CM4 An authorisation has occurred for every emergency change.
Security measures are in place to prevent, detect and remove malicious software.	10.4.1 Installation and regular update of malicious code detection and repair software to scan computers and media as a precautionary control or on a routine basis	DSS05.01 Protect against malware. Implement and maintain preventive, detective and corrective measures in place (especially up-to-date security patches and virus control) across the enterprise to protect information systems and technology from malware (e.g., viruses, worms, spyware, spam).	AV1 AV protection exists on all required assets. AV2 All AV signature updates that should have been made in the period have been made.
Security measures are in place to detect potential data breaches/data exfiltration transmissions and prevent them by monitoring, detecting and blocking sensitive data.	10.8.1 (a) Procedures designed to protect exchanged information from interception, copying, modification, misrouting and destruction 10.8.1 (g) Use of cryptographic techniques	DSS05: (2) Information processed on, stored on and transmitted by end point devices is protected. (5) Electronic information is properly secured when stored, transmitted or destroyed.	DLP1 Data loss prevention (DLP) protection exists on all required assets. DLP2 End point encryption exists on all required assets. DLP3 DLP protection exists on all network paths. DLP4 DLP alerts have been accurately recorded. DLP5 All DLP alerts that should have been disclosed (and actioned) have been disclosed and actioned.

Figure 4 — Proposed formal assertions for selected controls.

Defining Automated Tests

To continuously assess controls, rules need to be developed to test in real-time (or near-real-time) compliance with the previously mentioned formal assertions that are required to be made about the selected controls. The required tests can be classified into seven broad categories based on traditional audit processes or evidence types:

Asset management queries (where accurate), in place of physical examination of assets
Electronic transaction confirmations, in place of authenticated transaction documents, including verifying atomic elements of transactions
Electronic statement queries, in place of internal or external documentation
Re-performance of selected controls, using some form of automation
Observation (still a manual periodic test)
Analytical procedures, such as statistical analysis, comparisons with other internal or external data sets, and pattern-matching within transaction data
Automating collation of responses to inquiries such as control self-assessment surveys

The types of tests that could be employed in the case study example appear in figure 5.

Figure 5: Assertion test plan.
Proposed Assertion (refer to figure 4)	Test Type (refer to the seven test types noted previously)	Proposed Pass Condition for Test to Indicate a Strong Control
CM1. An authorisation has occurred for every change.	(2) Transaction confirmation	Percentage of changes with prior authorisation meeting the threshold for last two consecutive months
CM2. Testing has been completed for all changes prior to implementation.	(3) Statement (test results) query (6) Analytical procedure	High or very high confidence in testing for last two consecutive months, based on number of open defects
CM3. An approval has occurred for completeness of testing conducted.	(2) Transaction confirmation	Percentage of changes with testing approvals meeting threshold for last two consecutive months
CM4. An authorisation has occurred for every emergency change.	(2) Transaction confirmation	Percentage of emergency changes with authorisation meeting threshold for last two consecutive months
AV1. AV protection exists on all required assets.	(1) Asset management query	Percentage of required assets with AV protection meeting threshold for last two consecutive months
AV2. All AV signature updates that should have been made in the period have been made.	(2) Transaction confirmation	Percentage of assets with the latest AV signature meeting threshold for last two consecutive months
DLP1. DLP protection exists on all required assets.	(1) Asset management query	Percentage of required assets with DLP protection meeting threshold for last two consecutive months
DLP2. End point encryption exists on all required assets.	(1) Asset management query	Percentage of required assets with end point encryption meeting threshold for last two consecutive months
DLP3. DLP protection exists on all network paths.	(4) Re-performance (with test data) (7) Vendor control self-assessment	Result of weekly automated control tests to trigger DLP events, passing on two consecutive months (36 per annum)
DLP4. DLP alerts have been accurately recorded.	(4) Re-performance (with test data) (6) Analytical procedure	Result of weekly automated control tests to trigger DLP events, passing on two consecutive months (36 per annum)
DLP5. All DLP alerts that should have been disclosed (and actioned) have been disclosed and actioned.	(6) Analytical procedure	Statistical analysis of DLP alerts and corresponding incident actions to determine volatility of process

Figure 5 — Assertion test plan.

Generally, tests need to answer the question: What would the data look like if the control objective was met or was not met?

Asset management queries and transaction confirmation (type 1 and 2) tests can use existing or improved key risk indicators (KRIs) to provide what is described as a risk indicator continuous assurance (RICA) framework. Past audit report evidence can also be used to identify sources of data and applicable analytics. In this testing approach, a designated threshold being met in two or more consecutive months (or the majority of the time) may indicate a strong control, whereas the threshold not being met in two or more consecutive months may indicate a weak control.

Statement (or tabular data) tests (type 3) can use a belief function approach, in which evidence for and against an assertion is mathematically combined (or aggregated) to determine a result. In this approach, assurance levels are divided into five categories (very low, low, medium, high and very high) based on value ranges. For example, the strength of evidence supporting completeness of testing could be determined by ranges of test coverage or ranges of outstanding defect percentages.

Large data sets or complex behavioural controls may require analytical testing (type 6) to validate an assertion. This analysis may employ a risk score methodology or probability models to create an equal distribution of values 0 to 1 across all samples, with bands reflecting confidence in the assertion. The analysis may be based on:

Higher or lower than expected values
Expected or opposite to expected movement
Small or large changes from one period to the next
Process metrics
Erratic behaviour or volatility (variance) in the process

Assertions that need to be tested by subjective judgement (type 7, such as those obtained through control self-assessments by service managers or vendors) can be validated through the Delphi Method. In this approach, a more accurate consensus of control effectiveness is obtained through one or more rounds of anonymous self-assessments, which may be reviewed, and feedback provided by experts between rounds.

Planning for the implementation of any of the previously described automated tests needs to take into account likely difficulties such as obtaining data management approvals; data sourcing and aggregation lead times; the need for control domain expertise; technology acquisition and integration costs; and the need for information sharing and coordination among audit, risk and compliance functions.

Reporting

Figure 6 shows the governance and management processes associated with control assurance. Management monitors processes through mechanisms including KRIs, which are used to alert the business to potential control issues and are part of a continuous improvement cycle.

Figure 6: Control assurance processes — the control assurance programme evaluates control effectiveness, fed by issue management, risk and control self-assessment and the continuous monitoring of selected controls, all sitting above a plan, build, run and KRI monitoring cycle — **Figure 6** — Control assurance processes.

CCM takes selected KRIs and the results of other tests and analytics on processes and forms part of an overall control assurance program (CAP) in which the concerns over the monitored controls are validated before being prioritised and acted upon alongside issues identified by other periodic manual testing. Additional risk and key control deficiencies may also be identified through management risk and control self-assessments (RCSA) that form part of the program based on management knowledge gained through operating the plan-build-run-monitor cycle. Integrated issue management using a GRC platform facilitates digitisation, automation of alerts and management of remediation activities, once agreed upon by management.

Mature KRIs linked to formal assertions are continuously monitored and reported, automatically form part of the risk and control profile, and are integrated into daily management processes.

Other KRIs that may be subject to false positives are used in day-to-day management of the process in question and adjusted to a point where they can be relied upon for management self-assessment and continuous improvement of the process. As they mature, they can be incorporated in an expanded CCM regime.

Conclusion

This article provides guidance on the identification and prioritisation of controls for CCM implementation and introduces the need to transform COBIT (and other) management practices into formal assertions (in line with SAS 106) in order to facilitate objective automated testing. It defines the categories of testing available, maps a sample set of assertions to testing types and provides high-level guidance on applicable test rules.

Further work is needed to define formal assertions for the complete set of COBIT 5 management practices as a necessary precursor to the wider use of CCM within an IT risk context. This work ideally should occur with further development of COBIT 5 for Risk and other COBIT guidance from ISACA.

What to do with this

If the controls you report on are assessed by hand once a year, the first question is which of them could be tested from data you already hold — and which cannot, because the objective has never been written as something a test could decide. Working that out is part of the assurance and evidence workflow assessment: how the work actually runs today, what can be collected once instead of repeatedly, and what has to be rebuilt before any of it can run on its own.

Endnotes

Coderre, D.; Global Technology Audit Guide—Continuous Auditing: Implications for Assurance, Monitoring, and Risk Assessment, Institute of Internal Auditors, 2005
Vasarhelyi, M. A.; M. Alles; K. T. Williams; Continuous Assurance for the Now Economy, Institute of Chartered Accountants in Australia, 2010
MarFan, K.; IT Audit and Assurance Guideline G42, Continuous Assurance, ISACA, 2010, www.isaca.org/standards
Deloitte, Continuous Monitoring and Continuous Auditing: From Idea to Implementation, 2010
Gohil, J.; “Reduce Audit Time Using Automation, by Example,” presentation to ISACA Atlanta Chapter, Protiviti, 2013
Op cit Deloitte
Op cit Coderre
International Organization for Standardization and International Electrotechnical Commission, ISO/IEC 27002:2006, Information Technology—Security techniques—Code of practice for information security management, 2006
Op cit Vasarhelyi 2010
Op cit Standards Australia
Op cit Deloitte
Op cit MarFan
Op cit Vasarhelyi 2010
ISACA, 2009 CISA Review Manual, USA, 2008
Op cit Vasarhelyi 2010
ISACA, Relating the COSO Internal Control—Integrated Framework and COBIT, USA, 2014
American Institute of Certified Public Accountants (AICPA), SAS 106, “Audit Evidence,” February 2006
Op cit ISACA 2014
ISACA, IT Assurance Guide: Using COBIT, USA, 2007
Op cit Coderre
Majdalawieh, M.; S. Sahraoui; R. Barkhi; “Intra/Inter Process Continuous Auditing (IIPCA), Integrating CA Within an Enterprise System Environment”, Business Process Management Journal, 18 (2), 2012, p. 304–327
Vasarhelyi, M. A.; M. G. Alles; A. Kogan; “Principles of Analytic Monitoring for Continuous Assurance”, Journal of Emerging Technologies in Accounting, vol. 1, p. 1–21, 2004
Op cit Coderre
Nigrini, M. J.; A. J. Johnson; “Using Key Performance Indicators and Risk Measures in Continuous Monitoring”, Journal of Emerging Technologies in Accounting, vol. 5, 2008, p. 65–80
Op cit Vasarhelyi 2010
Op cit Dale
Mock, T. J.; A. Wright; R. P. Srivastava; “Audit Program Planning Using a Belief Function Framework”, Proceedings of the 1998 Deloitte & Touche University of Kansas Symposium on Auditing Problems, USA, 1998, p. 115–142
Op cit Nigrini
Alles, M. G.; A. Kogan; M. A. Vasarhelyi; “Putting Continuous Auditing Theory Into Practice: Lessons From Two Pilot Implementations”, Journal of Information Systems, 22 (2), 2008, p. 195–214
Op cit Vasarhelyi 2010
Vasarhelyi, M. A.; S. Romero; S. Kuenkaikaew; “Adopting Continuous Auditing/Continuous Monitoring in Internal Audit”, ISACA Journal, vol. 3, 2012, p. 1–5
Op cit Coderre
Schermann, M.; M. Wiesche; H. Krcmar; “The Role of Information Systems in Supporting Exploitative and Exploratory Management Control Activities”, Journal of Management Accounting Research, vol. 24, 2012, p. 31–59
Dale, J.; E. Chung Yee Wong; “Achieving Continuous IT Auditing: RICA”, ISACA Journal, vol. 6, 2009, p. 1–5
Op cit Coderre

First published as “A Practical Approach to Continuous Controls Monitoring”, ISACA Journal, Volume 2, 2015, Feature 1. Republished with permission. ISACA, COBIT and the ISACA Journal are trademarks of the Information Systems Audit and Control Association. COSO material is referenced from the Committee of Sponsoring Organizations of the Treadway Commission, and SAS 106 is published by the American Institute of Certified Public Accountants. Figures 2 to 5 are set as tables rather than reproduced as the original images; their content is unchanged. The author is the founder and CEO of MyRISK.